Re: Cloudstack manager - Unable to login after restart

[Jevgeni Zolotarjov]

That was it. I could never guess that myself.

But meanwhile I have managed to reinstall everything. And come
across another problem.
Now cloudstack-agent does not start because libvirtd service fails.
In libvirtd logs there is:
-- Unit libvirtd.service has begun starting up.
Apr 08 14:06:15 mtl1-apphst03.mt.pbt.com.mt libvirtd[4836]: 2020-04-08
14:06:15.675+: 4836: info : libvirt version: 4.5.0, package: 23.el7_7.6
(CentOS BuildSystem <http://bugs.centos.org>, 2020-03-17-23:39:10, x
Apr 08 14:06:15 mtl1-apphst03.mt.pbt.com.mt libvirtd[4836]: 2020-04-08
14:06:15.675+: 4836: info : hostname: mtl1-apphst03.mt.pbt.com.mt
Apr 08 14:06:15 mtl1-apphst03.mt.pbt.com.mt libvirtd[4836]: 2020-04-08
14:06:15.675+: 4836: error : virNetTLSContextCheckCertFile:112 : Cannot
read CA certificate '/etc/pki/CA/cacert.pem': No such file or direct
Apr 08 14:06:15 mtl1-apphst03.mt.pbt.com.mt systemd[1]: libvirtd.service:
main process exited, code=exited, status=6/NOTCONFIGURED
Apr 08 14:06:15 mtl1-apphst03.mt.pbt.com.mt systemd[1]: Failed to start
Virtualization daemon.
-- Subject: Unit libvirtd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit libvirtd.service has failed.
--
-- The result is failed.


Could it be another unintentional update problem?

On Wed, Apr 8, 2020 at 3:13 PM Andrija Panic 
wrote:

> you've upgrade the java-mysql-connector - downgrade back to 5.x if that's
> the case
>
>
> On Wed, 8 Apr 2020 at 10:08, Jevgeni Zolotarjov 
> wrote:
>
> > After restart of host machine (Centos 7.7) I am not able to login
> anymore.
> > Neither cloudstack starts any VMs.
> >
> > In the log I see following error right after login
> >
> >
> > 2020-04-08 08:05:49,979 DEBUG [c.c.u.AccountManagerImpl]
> > (qtp504527234-22:ctx-22182f59) (logid:ba27e53e) User: admin in domain 1
> has
> > successfully logged in
> > 2020-04-08 08:05:49,985 DEBUG [c.c.u.d.T.Transaction]
> > (qtp504527234-22:ctx-22182f59) (logid:ba27e53e) Rolling back the
> > transaction: Time = 4 Name =  qtp504527234-22; called by
> >
> >
> -TransactionLegacy.rollback:890-TransactionLegacy.removeUpTo:833-TransactionLegacy.close:657-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:174-ExposeInvocationInterceptor.invoke:92-ReflectiveMethodInvocation.proceed:185-JdkDynamicAopProxy.invoke:212-$Proxy129.persist:-1-ActionEventUtils.persistActionEvent:186-ActionEventUtils.onActionEvent:98-AccountManagerImpl.authenticateUser:2342
> > 2020-04-08 08:05:49,986 ERROR [c.c.a.ApiServlet]
> > (qtp504527234-22:ctx-22182f59) (logid:ba27e53e) unknown exception writing
> > api response
> > com.cloud.utils.exception.CloudRuntimeException: Problem with getting the
> > ec attribute
> > at
> > com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1454)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> Source)
> > at java.lang.reflect.Method.invoke(Unknown Source)
> > at
> >
> >
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> > at
> >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> > at
> >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > at
> >
> >
> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
> > at
> >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > at
> >
> >
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> > at
> >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> > at
> >
> >
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> > at com.sun.proxy.$Proxy129.persist(Unknown Source)
> > at
> >
> >
> com.cloud.event.ActionEventUtils.persistActionEvent(ActionEventUtils.java:186)
> > at
> > com.cloud.event.ActionEventUtils.onActionEvent(ActionEventUtils.java:98)
> > at
> >
> >
> com.cloud.user.AccountManagerImpl.authenticateUser(AccountManagerImpl.ja

Cloudstack manager - Unable to login after restart

[Jevgeni Zolotarjov]

After restart of host machine (Centos 7.7) I am not able to login anymore.
Neither cloudstack starts any VMs.

In the log I see following error right after login


2020-04-08 08:05:49,979 DEBUG [c.c.u.AccountManagerImpl]
(qtp504527234-22:ctx-22182f59) (logid:ba27e53e) User: admin in domain 1 has
successfully logged in
2020-04-08 08:05:49,985 DEBUG [c.c.u.d.T.Transaction]
(qtp504527234-22:ctx-22182f59) (logid:ba27e53e) Rolling back the
transaction: Time = 4 Name =  qtp504527234-22; called by
-TransactionLegacy.rollback:890-TransactionLegacy.removeUpTo:833-TransactionLegacy.close:657-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:174-ExposeInvocationInterceptor.invoke:92-ReflectiveMethodInvocation.proceed:185-JdkDynamicAopProxy.invoke:212-$Proxy129.persist:-1-ActionEventUtils.persistActionEvent:186-ActionEventUtils.onActionEvent:98-AccountManagerImpl.authenticateUser:2342
2020-04-08 08:05:49,986 ERROR [c.c.a.ApiServlet]
(qtp504527234-22:ctx-22182f59) (logid:ba27e53e) unknown exception writing
api response
com.cloud.utils.exception.CloudRuntimeException: Problem with getting the
ec attribute
at
com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1454)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy129.persist(Unknown Source)
at
com.cloud.event.ActionEventUtils.persistActionEvent(ActionEventUtils.java:186)
at
com.cloud.event.ActionEventUtils.onActionEvent(ActionEventUtils.java:98)
at
com.cloud.user.AccountManagerImpl.authenticateUser(AccountManagerImpl.java:2342)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy125.authenticateUser(Unknown Source)
at com.cloud.api.ApiServer.loginUser(ApiServer.java:1073)
at
com.cloud.api.auth.DefaultLoginAPIAuthenticatorCmd.authenticate(DefaultLoginAPIAuthenticatorCmd.java:172)
at
com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:214)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:130)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:127)
at com.cloud.api.ApiServlet.doPost(ApiServlet.java:94)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:706)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:852)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at

Re: unable to start management server after update 4.11.3 -> 4.13

[Jevgeni Zolotarjov]

Yes. I have made db backup.
Whats the correct way to downgrade cloudstack management in centos7?

On Sun, 29 Sep 2019, 20:06 Andrija Panic,  wrote:

> Right... You have probably added  custom guest OS type to the guest_os
> table before the upgrade, since I'm familiar this id 277 being a
> "duplicate".
>
> Remove from the guest_os table any custom added entries, since 4.13 adds a
> few guest OS types, with entries being created (in this guest_os table
> specifically) with IDs 277 up to 304 or so.
>
> In order to upgrade again, you will need to first restore tthe 4.11.3 db
> backup (you took right before the upgrade...), then delete from guest_os
> table anything with ID>275 (275 being the last ID in 4.11.3) and start mgmt
> (4.13) again.
>
> If you added  custom entries to other OS type related tables (i.e.
> hypervisor_guest_os if I remember the name from top of my head...), you
> might also want to remove those together with guest_os related entries...
>
> Ping here if any problems.
>
> Andrija
>
>
> On Sun, 29 Sep 2019, 16:11 Jevgeni Zolotarjov, 
> wrote:
>
> > I was running clean installation of 4.11.3
> >
> > Now after updating to 4.13 cloudstack-management does not start
> >
> > exception from management-server.log
> > 2019-09-29 14:05:57,058 WARN  [o.a.c.s.m.c.ResourceApplicationContext]
> > (main:null) (logid:) Exception encountered during context initialization
> -
> > cancelling refresh attempt:
> > org.springframework.context.ApplicationContextException: Failed to start
> > bean 'cloudStackLifeCycle'; nested exception is
> > com.cloud.utils.exception.CloudRuntimeException: Unable to upgrade the
> > database
> > 2019-09-29 14:05:57,059 WARN  [o.e.j.w.WebAppContext] (main:null)
> (logid:)
> > Failed startup of context o.e.j.w.WebAppContext@646be2c3
> >
> >
> {/client,file:///usr/share/cloudstack-management/webapp/,UNAVAILABLE}{/usr/share/cloudstack-management/webapp}
> > org.springframework.context.ApplicationContextException: Failed to start
> > bean 'cloudStackLifeCycle'; nested exception is
> > com.cloud.utils.exception.CloudRuntimeException: Unable to upgrade the
> > database
> > at
> >
> >
> org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:186)
> > at
> >
> >
> org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:52)
> > at
> >
> >
> org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:358)
> > at
> >
> >
> org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:159)
> > at
> >
> >
> org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123)
> > at
> >
> >
> org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:884)
> > at
> >
> >
> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:552)
> > at
> >
> >
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:145)
> > at
> >
> >
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:122)
> > at
> >
> >
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:245)
> > at
> >
> >
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
> > at
> >
> >
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:233)
> > at
> >
> >
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:117)
> > at
> >
> >
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:79)
> > at
> >
> >
> org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
> > at
> >
> >
> org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:71)
> > at
> >
> >
> org.apache.cloudstack.spring

unable to start management server after update 4.11.3 -> 4.13

[Jevgeni Zolotarjov]

I was running clean installation of 4.11.3

Now after updating to 4.13 cloudstack-management does not start

exception from management-server.log
2019-09-29 14:05:57,058 WARN  [o.a.c.s.m.c.ResourceApplicationContext]
(main:null) (logid:) Exception encountered during context initialization -
cancelling refresh attempt:
org.springframework.context.ApplicationContextException: Failed to start
bean 'cloudStackLifeCycle'; nested exception is
com.cloud.utils.exception.CloudRuntimeException: Unable to upgrade the
database
2019-09-29 14:05:57,059 WARN  [o.e.j.w.WebAppContext] (main:null) (logid:)
Failed startup of context o.e.j.w.WebAppContext@646be2c3
{/client,file:///usr/share/cloudstack-management/webapp/,UNAVAILABLE}{/usr/share/cloudstack-management/webapp}
org.springframework.context.ApplicationContextException: Failed to start
bean 'cloudStackLifeCycle'; nested exception is
com.cloud.utils.exception.CloudRuntimeException: Unable to upgrade the
database
at
org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:186)
at
org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:52)
at
org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:358)
at
org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:159)
at
org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123)
at
org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:884)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:552)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:145)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:122)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:245)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:233)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:117)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:79)
at
org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:71)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.(CloudStackSpringContext.java:58)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.(CloudStackSpringContext.java:62)
at
org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52)
at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:890)
at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532)
at
org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:853)
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344)
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1515)
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1477)
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785)
at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
at
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:273)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:115)
   

Re: "Command failed due to Internal Server Error" when stopping a VM

[Jevgeni Zolotarjov]

+1

Have experienced exactly the same problem. My host is centos7.
Would be interested to get the solution.

On Tue, 16 Jul 2019, 23:57 daniel bellido, 
wrote:

> Hello,
>
> I've done a fresh install of cloudstack 4.11.3 on 2 ubuntu 18.04 servers
> (1 server hosts the management server , the other hosts a KVM host).
> I've configured the cloud using the wizard.
> Everything works fine except that I receive "internal server errors" when
> I stop the VMs . From the UI , the status stays as "stopping'; so the
> workaround is to go to the DB and set the status as "stopped". Expunging
> the VM results also in a internal server error too.
>
> Looking at the management server logs, I can see these errors :
>
>
> 2019-07-16 22:49:59,967 DEBUG [o.a.c.n.t.BasicNetworkTopology]
> (AgentManager-Handler-2:null) (logid:) REMOVING DHCP ENTRY RULE
> 2019-07-16 22:49:59,967 DEBUG [o.a.c.n.t.BasicNetworkTopology]
> (AgentManager-Handler-2:null) (logid:) Applying dhcp entry in network
> Ntwk[204|Guest|6]
> 2019-07-16 22:49:59,971 WARN  [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-2:null) (logid:) Caught:
> java.lang.NullPointerException
> at
> org.apache.cloudstack.network.topology.BasicNetworkVisitor.visit(BasicNetworkVisitor.java:201)
> at com.cloud.network.rules.DhcpEntryRules.accept(DhcpEntryRules.java:64)
> at
> org.apache.cloudstack.network.topology.BasicNetworkTopology.applyRules(BasicNetworkTopology.java:390)
> at
> org.apache.cloudstack.network.topology.BasicNetworkTopology.removeDhcpEntry(BasicNetworkTopology.java:464)
> at
> com.cloud.network.element.VirtualRouterElement.removeDhcpEntry(VirtualRouterElement.java:972)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.cleanupNicDhcpDnsEntry(NetworkOrchestrator.java:2933)
> at com.cloud.vm.UserVmManagerImpl.finalizeStop(UserVmManagerImpl.java:4389)
> at
> com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1485)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handlePowerOffReportWithNoPendingJobsOnVM(VirtualMachineManagerImpl.java:4186)
> at
> com.cloud.vm.VirtualMachineManagerImpl.scanStalledVMInTransitionStateOnUpHost(VirtualMachineManagerImpl.java:4238)
> at
> com.cloud.vm.VirtualMachineManagerImpl.processCommands(VirtualMachineManagerImpl.java:3076)
> at
> com.cloud.agent.manager.AgentManagerImpl.handleCommands(AgentManagerImpl.java:317)
> at
> com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1296)
> at
> com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1383)
> at
> com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:712)
> at com.cloud.utils.nio.Task.call(Task.java:83)
> at com.cloud.utils.nio.Task.call(Task.java:29)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> 2019-07-16 22:51:00,004 ERROR [o.a.c.f.m.MessageDispatcher]
> (AgentManager-Handler-5:null) (logid:) Unexpected exception when calling
> com.cloud.vm.ClusteredVirtualMachineManagerImpl.HandlePowerStateReport
> java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.cloudstack.framework.messagebus.MessageDispatcher.dispatch(MessageDispatcher.java:75)
> at
> org.apache.cloudstack.framework.messagebus.MessageDispatcher.onPublishMessage(MessageDispatcher.java:45)
> at
> org.apache.cloudstack.framework.messagebus.MessageBusBase$SubscriptionNode.notifySubscribers(MessageBusBase.java:441)
> at
> org.apache.cloudstack.framework.messagebus.MessageBusBase.publish(MessageBusBase.java:178)
> at
> com.cloud.vm.VirtualMachinePowerStateSyncImpl.processReport(VirtualMachinePowerStateSyncImpl.java:147)
> at
> com.cloud.vm.VirtualMachinePowerStateSyncImpl.processHostVmStatePingReport(VirtualMachinePowerStateSyncImpl.java:68)
> at
> com.cloud.vm.VirtualMachineManagerImpl.processCommands(VirtualMachineManagerImpl.java:3071)
> at
> com.cloud.agent.manager.AgentManagerImpl.handleCommands(AgentManagerImpl.java:317)
> at
> com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1296)
> at
> com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1383)
> at
> com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:712)
> at com.cloud.utils.nio.Task.call(Task.java:83)
> at com.cloud.utils.nio.Task.call(Task.java:29)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> 

Re: failure after update 4.11.2 -> 4.11.3

[Jevgeni Zolotarjov]

I downgraded cloudstack to 4.11.2

But apparently as the result of previous  update libvirtd got updated. Now
it does not start with following error:

2019-07-04 08:39:43.761+: 27030: error :
virNetTLSContextLoadCredentials:671 : Unable to set x509 key and
certificate: /etc/pki/libvirt/private/serverkey.pem,
/etc/pki/libvirt/servercert.pem: The provided X.509 certificate list is not
sorted (in subject to issuer order)



How to resolve this problem?

On Thu, Jul 4, 2019 at 11:20 AM Rohit Yadav 
wrote:

> Hi Jevgeni,
>
>
> We're yet to update the release notes, fix the upgrade docs and announce
> the release which my colleague and RM Paul Angus will do soon.
>
>
> You need to setup the 4.11.3 systemvmtemplate and use the name/version
> suffix was `-4.11.3` while registering the template before the upgrade. (
> http://docs.cloudstack.apache.org/en/4.11.2.0/upgrading/upgrade/upgrade-4.11.html
> )
>
>
> Then use one of the upstream packages, link of which was shared on dev@
> to upgrade.
>
>
> Since you've already upgraded, you can downgrade the package to 4.11.2.0
> and remove the 4.11.3.0 version row in the cloud.version table. And then
> restart the management server. Then retry the systemvmtemplate (4.11.3)
> registration and upgrade to the available packages (as mentioned, we're yet
> to fix the notes and links).
>
>
> Regards,
>
> Rohit Yadav
>
> Software Architect, ShapeBlue
>
> https://www.shapeblue.com
>
> 
> From: Jevgeni Zolotarjov 
> Sent: Thursday, July 4, 2019 11:40:16 AM
> To: users@cloudstack.apache.org
> Subject: failure after update 4.11.2 -> 4.11.3
>
> Hello
>
> I followed installation steps as described here
>
> http://docs.cloudstack.apache.org/en/latest/upgrading/upgrade/upgrade-4.11.htm
>
> I made sure, that system VM templates were in READY state.
>
> But after update management server does not start
> ● cloudstack-management.service - CloudStack Management Server
>Loaded: loaded (/usr/lib/systemd/system/cloudstack-management.service;
> enabled; vendor preset: disabled)
>Active: active (running) since Thu 2019-07-04 06:00:45 GMT; 14s ago
>  Main PID: 13166 (java)
> Tasks: 62
>CGroup: /system.slice/cloudstack-management.service
>└─13166 /usr/bin/java
> -Djava.security.properties=/etc/cloudstack/management/java.security.ciphers
> -Djava.awt.headless=true -Dcom.sun.management.jmxremote=false -Xmx2G
> -XX:+HeapDumpOnOutOfMemoryErr...
>
> Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: at
>
> org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:65)
> Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: ... 47 more
> Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: Caused by:
> com.cloud.utils.exception.CloudRuntimeException: 4.11.3.0KVM SystemVm
> template not found. Cannot upgrade system Vms
> Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: at
>
> com.cloud.upgrade.dao.Upgrade41120to41130.updateSystemVmTemplates(Upgrade41120to41130.java:203)
> Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: at
>
> com.cloud.upgrade.dao.Upgrade41120to41130.performDataMigration(Upgrade41120to41130.java:60)
> Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: at
>
> com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:586)
> Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: ... 49 more
>
>
> How to fix this problem?
>
> rohit.ya...@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London  WC2E 9DPUK
> @shapeblue
>
>
>
>

Re: failure after update 4.11.2 -> 4.11.3

[Jevgeni Zolotarjov]

I checked that it is in READY state

Anyway. What to do now?

On Thu, Jul 4, 2019 at 10:03 AM Boris Stoyanov 
wrote:

> Hi Jevgeni,
>
> Did you made sure your 4.11.3 systemvm template is installed and usable
> before diving into upgrade?
>
> Looks like cloudstack cannot find a 4.11.3 template to carry on the
> template.
>
> Thanks,
> Bobby.
>
> On 4.07.19, 9:10, "Jevgeni Zolotarjov"  wrote:
>
> Hello
>
> I followed installation steps as described here
>
> http://docs.cloudstack.apache.org/en/latest/upgrading/upgrade/upgrade-4.11.htm
>
> I made sure, that system VM templates were in READY state.
>
> But after update management server does not start
> ● cloudstack-management.service - CloudStack Management Server
>Loaded: loaded
> (/usr/lib/systemd/system/cloudstack-management.service;
> enabled; vendor preset: disabled)
>Active: active (running) since Thu 2019-07-04 06:00:45 GMT; 14s ago
>  Main PID: 13166 (java)
> Tasks: 62
>CGroup: /system.slice/cloudstack-management.service
>└─13166 /usr/bin/java
>
> -Djava.security.properties=/etc/cloudstack/management/java.security.ciphers
> -Djava.awt.headless=true -Dcom.sun.management.jmxremote=false -Xmx2G
> -XX:+HeapDumpOnOutOfMemoryErr...
>
> Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: at
>
> org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:65)
> Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: ... 47 more
> Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: Caused by:
> com.cloud.utils.exception.CloudRuntimeException: 4.11.3.0KVM SystemVm
> template not found. Cannot upgrade system Vms
> Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: at
>
> com.cloud.upgrade.dao.Upgrade41120to41130.updateSystemVmTemplates(Upgrade41120to41130.java:203)
> Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: at
>
> com.cloud.upgrade.dao.Upgrade41120to41130.performDataMigration(Upgrade41120to41130.java:60)
> Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: at
>
> com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:586)
> Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: ... 49 more
>
>
> How to fix this problem?
>
>
>
> boris.stoya...@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London  WC2E 9DPUK
> @shapeblue
>
>
>
>

failure after update 4.11.2 -> 4.11.3

[Jevgeni Zolotarjov]

Hello

I followed installation steps as described here
http://docs.cloudstack.apache.org/en/latest/upgrading/upgrade/upgrade-4.11.htm

I made sure, that system VM templates were in READY state.

But after update management server does not start
● cloudstack-management.service - CloudStack Management Server
   Loaded: loaded (/usr/lib/systemd/system/cloudstack-management.service;
enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-07-04 06:00:45 GMT; 14s ago
 Main PID: 13166 (java)
Tasks: 62
   CGroup: /system.slice/cloudstack-management.service
   └─13166 /usr/bin/java
-Djava.security.properties=/etc/cloudstack/management/java.security.ciphers
-Djava.awt.headless=true -Dcom.sun.management.jmxremote=false -Xmx2G
-XX:+HeapDumpOnOutOfMemoryErr...

Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:65)
Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: ... 47 more
Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: Caused by:
com.cloud.utils.exception.CloudRuntimeException: 4.11.3.0KVM SystemVm
template not found. Cannot upgrade system Vms
Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: at
com.cloud.upgrade.dao.Upgrade41120to41130.updateSystemVmTemplates(Upgrade41120to41130.java:203)
Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: at
com.cloud.upgrade.dao.Upgrade41120to41130.performDataMigration(Upgrade41120to41130.java:60)
Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: at
com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:586)
Jul 04 06:00:59 mtl1-apphst03.mt.pbt.com.mt java[13166]: ... 49 more


How to fix this problem?

Re: cannot start system VMs: disaster after maintenance followup

[Jevgeni Zolotarjov]

This saga is to be continued.

"Security groups" was the correct keyword to resolve my problem.

Now all is in order and all VMs run.

One observation:
This guide here suggests to configure
/etc/libvirt/libvirtd.conf
and
/etc/sysconfig/libvirtd
under
Libvirt Configuration

But these files get overwritten every time  cloudstack-agent service is
restarted.
I think, there is inconsistency in this guide for sure.

But Now I face the other problem , which is probably related to correct
configuration of security groups, but maybe it a bug
We have following config

VM1 - running ngninx proxy

VM2  - server hosting webapp on 8080
VM3 - server hosting another webapp on 8080. This webapp is exposing is
connection over websocket - serving data stream

(1) client -> VM1:80/app -> VM2:8080/app
(2) client -> VM1:80/data -> VM3:8080/data

This was working fine before the reinstallation.
We found that it works, if we stop iptables.

But with iptables ON, (1) works, but (2) does not work - it gives
connection refused.
How can this be resolved?



On Fri, Mar 22, 2019 at 11:19 AM Dag Sonstebo 
wrote:

> Jevgeni - you've not provided any network troubleshooting findings - but
> this is all down to security groups so check these are in place and working.
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
>
> On 21/03/2019, 19:47, "Jevgeni Zolotarjov" 
> wrote:
>
> << Almost, but not completely.
>
> I am moving VMs one by one. They run and they get IP address from
> Cloudstack and get connected to network.
>
> But I cannot connect to VMs from other PCs in the same LAN. Ping is not
> responding too.
> What can be the problem here?
>
>
> On Thu, 21 Mar 2019, 21:23 Andrija Panic, 
> wrote:
>
> > Stick to 4.11.2 - 4.12 should be released withing few days
> officially.
> >
> > As for qemu-kvm-ev - yes, it's supposed to work - make sure to test
> new
> > versions obviously.
> >
> > Did you got your new installation running fine ?
> >
> > On Thu, 21 Mar 2019 at 19:26, Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> > wrote:
> >
> > > Andrija,
> > >
> > > I asked here in the group if its safe to try new version of KVM
> and got
> > > reply, that it works. It was back in September. So we installed it
> with
> > > yum install centos-release-qemu-ev
> > > yum install qemu-kvm-ev
> > >
> > > It worked fine ever since.
> > > But with new maintenance (yum update) apparently some breaking
> changes
> > were
> > > introduced.
> > > So, take care.
> > >
> > > Anyway, thanks. for help.
> > >
> > > As for your suggestion to use CS4.12. I haven't managed to find
> systemvm
> > > images for 4.12. Should I continue to use 4.11.12 systemvm?
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Mar 21, 2019 at 7:19 PM Andrija Panic <
> andrija.pa...@gmail.com>
> > > wrote:
> > >
> > > > Jevgeni, qemu-kvm 1.5.3 is the lastest official one for CentoS
> 7.6.XXX
> > > > (latest) which I'm running atm in my lab (just checked for
> update) -
> > how
> > > > did you manage to go to 2.0 (custom repo ?)
> > > >
> > > > On Thu, 21 Mar 2019 at 18:13, Ivan Kudryavtsev <
> > kudryavtsev...@bw-sw.com
> > > >
> > > > wrote:
> > > >
> > > > > Jevgeniy, simplest and the most obvious way is to flatten their
> > images
> > > > with
> > > > > "qemu-img convert", next import them as templates and recreate
> VMs
> > from
> > > > > those templates.
> > > > >
> > > > > чт, 21 мар. 2019 г. в 13:05, Jevgeni Zolotarjov <
> > > j.zolotar...@gmail.com
> > > > >:
> > > > >
> > > > > > What happened in the end was: qemu-kvm got updated to
> version 2.0
> > > > during
> > > > > > the maintenance.  We could not manage to make this KVM to
> work with
> > > > > > Cloudstack.
> > > > > > So we rolled back to version 1.5.3.
> > > > > >
> > > > > > And now we have clean c

Re: cannot start system VMs: disaster after maintenance followup

[Jevgeni Zolotarjov]

<< wrote:

> Stick to 4.11.2 - 4.12 should be released withing few days officially.
>
> As for qemu-kvm-ev - yes, it's supposed to work - make sure to test new
> versions obviously.
>
> Did you got your new installation running fine ?
>
> On Thu, 21 Mar 2019 at 19:26, Jevgeni Zolotarjov 
> wrote:
>
> > Andrija,
> >
> > I asked here in the group if its safe to try new version of KVM and got
> > reply, that it works. It was back in September. So we installed it with
> > yum install centos-release-qemu-ev
> > yum install qemu-kvm-ev
> >
> > It worked fine ever since.
> > But with new maintenance (yum update) apparently some breaking changes
> were
> > introduced.
> > So, take care.
> >
> > Anyway, thanks. for help.
> >
> > As for your suggestion to use CS4.12. I haven't managed to find systemvm
> > images for 4.12. Should I continue to use 4.11.12 systemvm?
> >
> >
> >
> >
> >
> >
> > On Thu, Mar 21, 2019 at 7:19 PM Andrija Panic 
> > wrote:
> >
> > > Jevgeni, qemu-kvm 1.5.3 is the lastest official one for CentoS 7.6.XXX
> > > (latest) which I'm running atm in my lab (just checked for update) -
> how
> > > did you manage to go to 2.0 (custom repo ?)
> > >
> > > On Thu, 21 Mar 2019 at 18:13, Ivan Kudryavtsev <
> kudryavtsev...@bw-sw.com
> > >
> > > wrote:
> > >
> > > > Jevgeniy, simplest and the most obvious way is to flatten their
> images
> > > with
> > > > "qemu-img convert", next import them as templates and recreate VMs
> from
> > > > those templates.
> > > >
> > > > чт, 21 мар. 2019 г. в 13:05, Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com
> > > >:
> > > >
> > > > > What happened in the end was: qemu-kvm got updated to version 2.0
> > > during
> > > > > the maintenance.  We could not manage to make this KVM to work with
> > > > > Cloudstack.
> > > > > So we rolled back to version 1.5.3.
> > > > >
> > > > > And now we have clean cloudstack fully operational. We can create
> new
> > > VMs
> > > > > and it works. I am almost happy.
> > > > >
> > > > > Now question - how do I get my old VMs to work, considering I have
> > only
> > > > > their volumes?
> > > > >
> > > > > On Thu, Mar 21, 2019 at 6:24 PM Andrija Panic <
> > andrija.pa...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Just replace the URL for systemVM template from 4.11.1 with
> 4.11.2
> > > > (there
> > > > > > is a PR for this now).
> > > > > >
> > > > > > On Thu, 21 Mar 2019 at 16:53, Andrija Panic <
> > andrija.pa...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Please use the one, updated specifically for CentOS 7 -
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst
> > > > > > >
> > > > > > > And please avoid collocating KVM and MGMT on same server
> > > (especially
> > > > in
> > > > > > > any production-like system)
> > > > > > >
> > > > > > > Please let me know if the guide above gives you problem - we
> had
> > > > > multiple
> > > > > > > users explicitly following it - and successfully installed
> (with
> > > some
> > > > > > minor
> > > > > > > modification, which we committed back to that guide).
> > > > > > >
> > > > > > > Thanks
> > > > > > > Andrija
> > > > > > >
> > > > > > > On Thu, 21 Mar 2019 at 16:34, Jevgeni Zolotarjov <
> > > > > j.zolotar...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> OS management - centos 7 (1810)
> > > > > > >> OS hypervisor - centos 7 (1810)
> > > > > > >>
> > > > > > >> Basic zone - yes
> > > > > > >> I 

Re: cannot start system VMs: disaster after maintenance followup

[Jevgeni Zolotarjov]

Andrija,

I asked here in the group if its safe to try new version of KVM and got
reply, that it works. It was back in September. So we installed it with
yum install centos-release-qemu-ev
yum install qemu-kvm-ev

It worked fine ever since.
But with new maintenance (yum update) apparently some breaking changes were
introduced.
So, take care.

Anyway, thanks. for help.

As for your suggestion to use CS4.12. I haven't managed to find systemvm
images for 4.12. Should I continue to use 4.11.12 systemvm?






On Thu, Mar 21, 2019 at 7:19 PM Andrija Panic 
wrote:

> Jevgeni, qemu-kvm 1.5.3 is the lastest official one for CentoS 7.6.XXX
> (latest) which I'm running atm in my lab (just checked for update) - how
> did you manage to go to 2.0 (custom repo ?)
>
> On Thu, 21 Mar 2019 at 18:13, Ivan Kudryavtsev 
> wrote:
>
> > Jevgeniy, simplest and the most obvious way is to flatten their images
> with
> > "qemu-img convert", next import them as templates and recreate VMs from
> > those templates.
> >
> > чт, 21 мар. 2019 г. в 13:05, Jevgeni Zolotarjov  >:
> >
> > > What happened in the end was: qemu-kvm got updated to version 2.0
> during
> > > the maintenance.  We could not manage to make this KVM to work with
> > > Cloudstack.
> > > So we rolled back to version 1.5.3.
> > >
> > > And now we have clean cloudstack fully operational. We can create new
> VMs
> > > and it works. I am almost happy.
> > >
> > > Now question - how do I get my old VMs to work, considering I have only
> > > their volumes?
> > >
> > > On Thu, Mar 21, 2019 at 6:24 PM Andrija Panic  >
> > > wrote:
> > >
> > > > Just replace the URL for systemVM template from 4.11.1 with 4.11.2
> > (there
> > > > is a PR for this now).
> > > >
> > > > On Thu, 21 Mar 2019 at 16:53, Andrija Panic  >
> > > > wrote:
> > > >
> > > > > Please use the one, updated specifically for CentOS 7 -
> > > > >
> > > >
> > >
> >
> https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst
> > > > >
> > > > > And please avoid collocating KVM and MGMT on same server
> (especially
> > in
> > > > > any production-like system)
> > > > >
> > > > > Please let me know if the guide above gives you problem - we had
> > > multiple
> > > > > users explicitly following it - and successfully installed (with
> some
> > > > minor
> > > > > modification, which we committed back to that guide).
> > > > >
> > > > > Thanks
> > > > > Andrija
> > > > >
> > > > > On Thu, 21 Mar 2019 at 16:34, Jevgeni Zolotarjov <
> > > j.zolotar...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >> OS management - centos 7 (1810)
> > > > >> OS hypervisor - centos 7 (1810)
> > > > >>
> > > > >> Basic zone - yes
> > > > >> I am following this quide
> > > > >>
> > > > >>
> > > >
> > >
> >
> http://docs.cloudstack.apache.org/en/4.11.2.0/quickinstallationguide/qig.html
> > > > >>
> > > > >> Right now from scratch - management ans hypervisor on the same
> > machine
> > > > >> qemu - version 1.5.3
> > > > >> libvirt - libvirt version: 4.5.0, package: 10.el7_6.6
> > > > >>
> > > > >> Basically - everything out of the box of clean centos install
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Thu, Mar 21, 2019 at 5:08 PM Andrija Panic <
> > > andrija.pa...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hey Jevgeni,
> > > > >> >
> > > > >> > what OS mgmt, what OS hypervisor, what qemu/libvirt versions -
> > still
> > > > in
> > > > >> > Basic Zone, SG ?
> > > > >> >
> > > > >> > Andrija
> > > > >> >
> > > > >> > On Thu, 21 Mar 2019 at 13:06, Jevgeni Zolotarjov <
> > > > >> j.zolotar...@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > I reinstalled

Re: cannot start system VMs: disaster after maintenance followup

[Jevgeni Zolotarjov]

What happened in the end was: qemu-kvm got updated to version 2.0 during
the maintenance.  We could not manage to make this KVM to work with
Cloudstack.
So we rolled back to version 1.5.3.

And now we have clean cloudstack fully operational. We can create new VMs
and it works. I am almost happy.

Now question - how do I get my old VMs to work, considering I have only
their volumes?

On Thu, Mar 21, 2019 at 6:24 PM Andrija Panic 
wrote:

> Just replace the URL for systemVM template from 4.11.1 with 4.11.2 (there
> is a PR for this now).
>
> On Thu, 21 Mar 2019 at 16:53, Andrija Panic 
> wrote:
>
> > Please use the one, updated specifically for CentOS 7 -
> >
> https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst
> >
> > And please avoid collocating KVM and MGMT on same server (especially in
> > any production-like system)
> >
> > Please let me know if the guide above gives you problem - we had multiple
> > users explicitly following it - and successfully installed (with some
> minor
> > modification, which we committed back to that guide).
> >
> > Thanks
> > Andrija
> >
> > On Thu, 21 Mar 2019 at 16:34, Jevgeni Zolotarjov  >
> > wrote:
> >
> >> OS management - centos 7 (1810)
> >> OS hypervisor - centos 7 (1810)
> >>
> >> Basic zone - yes
> >> I am following this quide
> >>
> >>
> http://docs.cloudstack.apache.org/en/4.11.2.0/quickinstallationguide/qig.html
> >>
> >> Right now from scratch - management ans hypervisor on the same machine
> >> qemu - version 1.5.3
> >> libvirt - libvirt version: 4.5.0, package: 10.el7_6.6
> >>
> >> Basically - everything out of the box of clean centos install
> >>
> >>
> >>
> >>
> >> On Thu, Mar 21, 2019 at 5:08 PM Andrija Panic 
> >> wrote:
> >>
> >> > Hey Jevgeni,
> >> >
> >> > what OS mgmt, what OS hypervisor, what qemu/libvirt versions - still
> in
> >> > Basic Zone, SG ?
> >> >
> >> > Andrija
> >> >
> >> > On Thu, 21 Mar 2019 at 13:06, Jevgeni Zolotarjov <
> >> j.zolotar...@gmail.com>
> >> > wrote:
> >> >
> >> > > I reinstalled cloudstack from scratch - everything
> >> > >
> >> > > But looks like I hit the same wall now
> >> > >
> >> > > In the last step of installation it cannot create system VMs.
> >> > >
> >> > > service libvirtd status -l
> >> > > gives me
> >> > > 
> >> > > ● libvirtd.service - Virtualization daemon
> >> > >Loaded: loaded (/usr/lib/systemd/system/libvirtd.service;
> enabled;
> >> > > vendor preset: enabled)
> >> > >Active: active (running) since Thu 2019-03-21 11:45:00 GMT; 18min
> >> ago
> >> > >  Docs: man:libvirtd(8)
> >> > >https://libvirt.org
> >> > >  Main PID: 537 (libvirtd)
> >> > > Tasks: 20 (limit: 32768)
> >> > >CGroup: /system.slice/libvirtd.service
> >> > >├─  537 /usr/sbin/libvirtd -l
> >> > >├─12206 /usr/sbin/dnsmasq
> >> > > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
> >> > > --dhcp-script=/usr/libexec/libvirt_leaseshelper
> >> > >└─12207 /usr/sbin/dnsmasq
> >> > > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
> >> > > --dhcp-script=/usr/libexec/libvirt_leaseshelper
> >> > >
> >> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]:
> 2019-03-21
> >> > > 11:45:01.168+: 566: info : libvirt version: 4.5.0, package:
> >> > 10.el7_6.6
> >> > > (CentOS BuildSystem <http://bugs.centos.org>, 2019-03-14-10:21:47,
> >> > > x86-01.bsys.centos.org)
> >> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]:
> 2019-03-21
> >> > > 11:45:01.168+: 566: info : hostname:
> mtl1-apphst03.mt.pbt.com.mt
> >> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]:
> 2019-03-21
> >> > > 11:45:01.168+: 566: error : virFirewallApplyRuleDirect:709 :
> >> internal
> >> > > error: Failed to apply firewall rules /usr/sbin/iptables -w --table
> >> nat
> >> > > --insert POSTROUTING --source 192.168.122.0/24 '

Re: cannot start system VMs: disaster after maintenance followup

[Jevgeni Zolotarjov]

OS management - centos 7 (1810)
OS hypervisor - centos 7 (1810)

Basic zone - yes
I am following this quide
http://docs.cloudstack.apache.org/en/4.11.2.0/quickinstallationguide/qig.html

Right now from scratch - management ans hypervisor on the same machine
qemu - version 1.5.3
libvirt - libvirt version: 4.5.0, package: 10.el7_6.6

Basically - everything out of the box of clean centos install




On Thu, Mar 21, 2019 at 5:08 PM Andrija Panic 
wrote:

> Hey Jevgeni,
>
> what OS mgmt, what OS hypervisor, what qemu/libvirt versions - still in
> Basic Zone, SG ?
>
> Andrija
>
> On Thu, 21 Mar 2019 at 13:06, Jevgeni Zolotarjov 
> wrote:
>
> > I reinstalled cloudstack from scratch - everything
> >
> > But looks like I hit the same wall now
> >
> > In the last step of installation it cannot create system VMs.
> >
> > service libvirtd status -l
> > gives me
> > 
> > ● libvirtd.service - Virtualization daemon
> >Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
> > vendor preset: enabled)
> >Active: active (running) since Thu 2019-03-21 11:45:00 GMT; 18min ago
> >  Docs: man:libvirtd(8)
> >https://libvirt.org
> >  Main PID: 537 (libvirtd)
> > Tasks: 20 (limit: 32768)
> >CGroup: /system.slice/libvirtd.service
> >├─  537 /usr/sbin/libvirtd -l
> >├─12206 /usr/sbin/dnsmasq
> > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
> > --dhcp-script=/usr/libexec/libvirt_leaseshelper
> >└─12207 /usr/sbin/dnsmasq
> > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
> > --dhcp-script=/usr/libexec/libvirt_leaseshelper
> >
> > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
> > 11:45:01.168+: 566: info : libvirt version: 4.5.0, package:
> 10.el7_6.6
> > (CentOS BuildSystem <http://bugs.centos.org>, 2019-03-14-10:21:47,
> > x86-01.bsys.centos.org)
> > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
> > 11:45:01.168+: 566: info : hostname: mtl1-apphst03.mt.pbt.com.mt
> > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
> > 11:45:01.168+: 566: error : virFirewallApplyRuleDirect:709 : internal
> > error: Failed to apply firewall rules /usr/sbin/iptables -w --table nat
> > --insert POSTROUTING --source 192.168.122.0/24 '!' --destination
> > 192.168.122.0/24 --jump MASQUERADE: iptables v1.4.21: can't initialize
> > iptables table `nat': Table does not exist (do you need to insmod?)
> > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: Perhaps
> > iptables
> > or your kernel needs to be upgraded.
> > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt dnsmasq[12206]: read
> > /etc/hosts
> > - 4 addresses
> > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt dnsmasq[12206]: read
> > /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
> > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt dnsmasq-dhcp[12206]: read
> > /var/lib/libvirt/dnsmasq/default.hostsfile
> > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
> > 11:45:01.354+: 566: warning : virSecurityManagerNew:189 : Configured
> > security driver "none" disables default policy to create confined guests
> > Mar 21 11:49:57 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
> > 11:49:57.354+: 542: warning : qemuDomainObjTaint:7521 : Domain id=2
> > name='s-1-VM' uuid=1a06d3a7-4e3f-4cba-912f-74ae24569bac is tainted:
> > high-privileges
> > Mar 21 11:49:59 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
> > 11:49:59.402+: 540: warning : qemuDomainObjTaint:7521 : Domain id=3
> > name='v-2-VM' uuid=af2a8342-cd9b-4b55-ba12-480634a31d65 is tainted:
> > high-privileges
> >
> >
> > What can be done about that ?
> >
>
>
> --
>
> Andrija Panić
>

cannot start system VMs: disaster after maintenance followup

[Jevgeni Zolotarjov]

I reinstalled cloudstack from scratch - everything

But looks like I hit the same wall now

In the last step of installation it cannot create system VMs.

service libvirtd status -l
gives me

● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
vendor preset: enabled)
   Active: active (running) since Thu 2019-03-21 11:45:00 GMT; 18min ago
 Docs: man:libvirtd(8)
   https://libvirt.org
 Main PID: 537 (libvirtd)
Tasks: 20 (limit: 32768)
   CGroup: /system.slice/libvirtd.service
   ├─  537 /usr/sbin/libvirtd -l
   ├─12206 /usr/sbin/dnsmasq
--conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
--dhcp-script=/usr/libexec/libvirt_leaseshelper
   └─12207 /usr/sbin/dnsmasq
--conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
--dhcp-script=/usr/libexec/libvirt_leaseshelper

Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
11:45:01.168+: 566: info : libvirt version: 4.5.0, package: 10.el7_6.6
(CentOS BuildSystem , 2019-03-14-10:21:47,
x86-01.bsys.centos.org)
Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
11:45:01.168+: 566: info : hostname: mtl1-apphst03.mt.pbt.com.mt
Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
11:45:01.168+: 566: error : virFirewallApplyRuleDirect:709 : internal
error: Failed to apply firewall rules /usr/sbin/iptables -w --table nat
--insert POSTROUTING --source 192.168.122.0/24 '!' --destination
192.168.122.0/24 --jump MASQUERADE: iptables v1.4.21: can't initialize
iptables table `nat': Table does not exist (do you need to insmod?)
Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: Perhaps iptables
or your kernel needs to be upgraded.
Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt dnsmasq[12206]: read /etc/hosts
- 4 addresses
Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt dnsmasq[12206]: read
/var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt dnsmasq-dhcp[12206]: read
/var/lib/libvirt/dnsmasq/default.hostsfile
Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
11:45:01.354+: 566: warning : virSecurityManagerNew:189 : Configured
security driver "none" disables default policy to create confined guests
Mar 21 11:49:57 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
11:49:57.354+: 542: warning : qemuDomainObjTaint:7521 : Domain id=2
name='s-1-VM' uuid=1a06d3a7-4e3f-4cba-912f-74ae24569bac is tainted:
high-privileges
Mar 21 11:49:59 mtl1-apphst03.mt.pbt.com.mt libvirtd[537]: 2019-03-21
11:49:59.402+: 540: warning : qemuDomainObjTaint:7521 : Domain id=3
name='v-2-VM' uuid=af2a8342-cd9b-4b55-ba12-480634a31d65 is tainted:
high-privileges


What can be done about that ?

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

It started with 4.10 and then gradually upgraded with all stops, when new
releases were available.


>>> Why do you have 3 zones in this installation - what is the setup ?
>>> SSVM and CPVM (for whatever zone) are failing to be created...
Its a result of attempts to create new zone and somehow move VMs to this
new zone. These all are unsuccessful attempts.
Before problem started there was 1 zone and There should be just 1 zone in
reality.


>>> yes, the VR can't be started, it get's timeout - in AGENT logs, I see
that
>>> it attemps to create a volume on primary storage...
I guess this is the root cause. I checked, and primary storage is
accessible via NFS share on both hosts. How to troubleshoot it?


On Wed, Mar 20, 2019 at 6:29 PM Andrija Panic 
wrote:

> Hi,
>
> 2019-03-20 06:41:50,446 INFO  [c.c.u.DatabaseUpgradeChecker] (main:null)
> (logid:) DB version = 4.10.0.0 Code Version = 4.11.2.0
> 2019-03-20 06:41:50,447 DEBUG [c.c.u.DatabaseUpgradeChecker] (main:null)
> (logid:) Running upgrade Upgrade41000to41100 to upgrade from
> 4.10.0.0-4.11.0.0 to 4.11.0.0
> fails due to
> java.sql.SQLException: Error on rename of './cloud/ldap_trust_map' to
> './cloud/#sql2-2f01-13d' (errno: 152)
>
> Then later...
>
> com.cloud.exception.InsufficientServerCapacityException: Unable to create a
> deployment for VM[SecondaryStorageVm|s-734-VM]Scope=interface
> com.cloud.dc.DataCenter; id=3
> com.cloud.exception.InsufficientServerCapacityException: Unable to create a
> deployment for VM[ConsoleProxy|v-733-VM]Scope=interface
> com.cloud.dc.DataCenter; id=3
>
> 2019-03-20 15:02:39,113 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 1 is ready to launch
> secondary storage VM
> 2019-03-20 15:02:39,117 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 2 is not ready to launch
> secondary storage VM yet
> 2019-03-20 15:02:39,122 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 3 is ready to launch
> secondary storage VM
>
> so did you start with clean 4.11.2 install, or was it upgraded one - I see
> in logs an upgrade from DB version 4.10 to 4.11 was tried and failed...
> Why do you have 3 zones in this installation - what is the setup ?
> SSVM and CPVM (for whatever zone) are failing to be created...
>
> yes, the VR can't be started, it get's timeout - in AGENT logs, I see that
> it attemps to create a volume on primary storage...
>
>
> Also, for SSVM I got this one...
> 2019-03-20 14:38:09,227 DEBUG [c.c.d.FirstFitPlanner]
> (Work-Job-Executor-96:ctx-04c5c9f2 job-5120/job-6960 ctx-fde3d4d7)
> (logid:49483c7a) No clusters found having a host with enough capacity,
> returning.
>
> Andrija
>
> On Wed, 20 Mar 2019 at 16:39, Jevgeni Zolotarjov 
> wrote:
>
> > Basic Zone - Yes
> >
> > router has been actually started/created on KVM side - not created, not
> > started. Thats the main problem, I guess
> >
> > agent.log
> > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
> >
> > management log
> > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> >
> > >> Can you confirm your zone/pod/cluster/hosts are all in Enabled state,
> > i.e.
> > YES, all green
> >
> > >> Can you connect your both KVM hosts can access/mount both Primary and
> > Secondary Storage
> > YES. Double checked
> >
> > >>>Can you also explain your infrastructure - you said you have two hosts
> > only, where does CloudStack management run?
> > 2 hosts:
> > host1: 192.168.1.14
> > host2: 192.168.1.5
> >
> > Servers are standing next to each other - connected to the same switch
> > Management server runs on the same physical server with host1
> >
> > I noticed, that Virtual router gets created after I try to start any of
> the
> > existing guest VM
> > Here are logs
> > management:
> > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> >
> > agent on host1:
> > https://drive.google.com/open?id=1u8YHYIuyU2MA2UKY7G5z7q8p5XxU1zsy
> >
> > agent on host2:
> > https://drive.google.com/open?id=1YzkCL-FmTgPva-QHHp5vTM5Nb3qAXxz4
> >
> > But this virtual router stays in Starting state forever and hence VMs do
> > not start either.
> >
> > On Wed, Mar 20, 2019 at 2:49 PM Andrija Panic 
> > wrote:
> >
> > > Just to confirm, you are using Basic Zone in CloudStack, right ?
> > >
> > > Can you confirm that router has been actually started/created on KVM
> > side,
> > >

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

Basic Zone - Yes

router has been actually started/created on KVM side - not created, not
started. Thats the main problem, I guess

agent.log
https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-

management log
https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A

>> Can you confirm your zone/pod/cluster/hosts are all in Enabled state,
i.e.
YES, all green

>> Can you connect your both KVM hosts can access/mount both Primary and
Secondary Storage
YES. Double checked

>>>Can you also explain your infrastructure - you said you have two hosts
only, where does CloudStack management run?
2 hosts:
host1: 192.168.1.14
host2: 192.168.1.5

Servers are standing next to each other - connected to the same switch
Management server runs on the same physical server with host1

I noticed, that Virtual router gets created after I try to start any of the
existing guest VM
Here are logs
management:
https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A

agent on host1:
https://drive.google.com/open?id=1u8YHYIuyU2MA2UKY7G5z7q8p5XxU1zsy

agent on host2:
https://drive.google.com/open?id=1YzkCL-FmTgPva-QHHp5vTM5Nb3qAXxz4

But this virtual router stays in Starting state forever and hence VMs do
not start either.

On Wed, Mar 20, 2019 at 2:49 PM Andrija Panic 
wrote:

> Just to confirm, you are using Basic Zone in CloudStack, right ?
>
> Can you confirm that router has been actually started/created on KVM side,
> again, as requested please post logs (mgmt and agent - and note the time
> around which you tried to start VR last time it partially succeeded) - we
> can't guess what went wrong without logs.
>
> I would push more effort solving this one, instead of reinstalling - you
> might hit the issue again and then it's no good.
>
> Can you confirm your zone/pod/cluster/hosts are all in Enabled state, i.e.
> not disabled and hosts connected AND both SSVM and CPVM are
> connectedUP/green
> Is your dashboard in GUI all green - no issues there ?
> Can you connect your both KVM hosts can access/mount both Primary and
> Secondary Storage
>
> On Wed, 20 Mar 2019 at 13:15, Jevgeni Zolotarjov 
> wrote:
>
> > After dozen of attempts, the Virtual Router could finally be recreated.
> But
> > its in eternal Starting status, and console prompts it required upgrade
> and
> > Version is UNKNOWN
> >
> > It does not resolve the problem, I cannot move further form this point.
> > Any hints?
> >
> > Or I am condemned to do reinstall cloudstack from scratch?
> >
> > On Wed, Mar 20, 2019 at 11:08 AM Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com>
> > wrote:
> >
> > > Under this defaultGuestNetwork, I go to Virtual Appliances. There is no
> > > VMS - "no data to show"
> > >
> > > I dont have any network, other than this single default one.
> > >
> > > I've tried adding new network - Add guest network. But I am not able to
> > do
> > > so, cause in the wizard popup, it offers empty dropdown with Zones
> > > selection. And this wizard doesnt not allow to go further without
> > selecting
> > > Zone
> > >
> > > On Wed, Mar 20, 2019 at 10:28 AM Andrija Panic <
> andrija.pa...@gmail.com>
> > > wrote:
> > >
> > >> you need to delete/remove all VMs inside this network (tick the
> > "Expunge"
> > >> button during VM deletion - if you want to really delete the VMs) in
> > order
> > >> to be able to delete the network - OR simply attach this VM to another
> > >> network, make this new network a DEFAULT one (NIC that is...), and
> then
> > >> detach from old network - and then effectively your VM was "removed"
> > from
> > >> old network - after this you should be able to delete the old
> network. I
> > >> assume some DB incosistencies perhaps, being the reason you can not
> > >> restart
> > >> the network.
> > >>
> > >> Did you try restarting some other Network - or deploying a new
> network,
> > >> spin a VM in it, then again try to restart this new network - does it
> > >> work ?
> > >>
> > >> Andrija
> > >>
> > >> On Wed, 20 Mar 2019 at 08:58, Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com>
> > >> wrote:
> > >>
> > >> > >>>Stop mgmt,
> > >> > >>>Stop all agents
> > >> > >>>Restart libvirtd (and check libvirt logs afterwards)
> > >> > >>>Start agents
> > >> > >>>Start mgmt.
> > >> >
> &g

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

After dozen of attempts, the Virtual Router could finally be recreated. But
its in eternal Starting status, and console prompts it required upgrade and
Version is UNKNOWN

It does not resolve the problem, I cannot move further form this point.
Any hints?

Or I am condemned to do reinstall cloudstack from scratch?

On Wed, Mar 20, 2019 at 11:08 AM Jevgeni Zolotarjov 
wrote:

> Under this defaultGuestNetwork, I go to Virtual Appliances. There is no
> VMS - "no data to show"
>
> I dont have any network, other than this single default one.
>
> I've tried adding new network - Add guest network. But I am not able to do
> so, cause in the wizard popup, it offers empty dropdown with Zones
> selection. And this wizard doesnt not allow to go further without selecting
> Zone
>
> On Wed, Mar 20, 2019 at 10:28 AM Andrija Panic 
> wrote:
>
>> you need to delete/remove all VMs inside this network (tick the "Expunge"
>> button during VM deletion - if you want to really delete the VMs) in order
>> to be able to delete the network - OR simply attach this VM to another
>> network, make this new network a DEFAULT one (NIC that is...), and then
>> detach from old network - and then effectively your VM was "removed" from
>> old network - after this you should be able to delete the old network. I
>> assume some DB incosistencies perhaps, being the reason you can not
>> restart
>> the network.
>>
>> Did you try restarting some other Network - or deploying a new network,
>> spin a VM in it, then again try to restart this new network - does it
>> work ?
>>
>> Andrija
>>
>> On Wed, 20 Mar 2019 at 08:58, Jevgeni Zolotarjov 
>> wrote:
>>
>> > >>>Stop mgmt,
>> > >>>Stop all agents
>> > >>>Restart libvirtd (and check libvirt logs afterwards)
>> > >>>Start agents
>> > >>>Start mgmt.
>> >
>> > I did that numerous time. Nothing really suspicious
>> > I can see that systems VMs are running - both in cloudstack console and
>> > with virsh list -all
>> >
>> > It is apparently problem with network.
>> > Is there a way to force recreation of defaultGuestNetwork? or force
>> > recreation of Virtual Router.
>> > I am unable to delete network, which is supposed to rebuild network with
>> > its router. Thats the issue
>> >
>> > The issue with libvirtd was, that eventually at some point it was
>> updated
>> > during 4 months of running, and not rebooted. It still worked. We had to
>> > add listen_tcp = 1 for libvirtd to start working again.
>> >
>> > On Wed, Mar 20, 2019 at 9:49 AM Andrija Panic 
>> > wrote:
>> >
>> > > As Sergey suggested... but i would also verify no libvirt issues or
>> > storage
>> > > pool issues - so perhaps:
>> > >
>> > > Stop mgmt,
>> > > Stop all agents
>> > > Restart libvirtd (and check libvirt logs afterwards)
>> > > Start agents
>> > > Start mgmt.
>> > >
>> > > What was originally issue with libvirtd ?
>> > > That sounds fishy to me...
>> > >
>> > > Andrija
>> > >
>> > > On Wed, Mar 20, 2019, 02:15 Sergey Levitskiy 
>> > wrote:
>> > >
>> > > > select * from networks where removed is null;
>> > > > select * from vm_instance where id=87;
>> > > > select id,name from vm_instance where name like 'r%' and removed is
>> > null;
>> > > >
>> > > > Basically since the network offering is not redundant this error is
>> > only
>> > > > thrown when there is no router associated with your network. Usually
>> > > > management server restart tries to implement network again. Please
>> > > restart
>> > > > management server, save and share management server log.
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On 3/19/19, 3:31 PM, "Jevgeni Zolotarjov" 
>> > > wrote:
>> > > >
>> > > > >>>>Check network_offering table for  value in column
>> > > > redundant_router_service  for the network offering you use.
>> > > > in table network_offering_table all records have
>> > > > redundant_router_service =
>> > > > 0
>> > > >
>> > > > Can you also run the following:
>> > > > >>>select n

How to reimport instances from failed cluster

[Jevgeni Zolotarjov]

I've come across a problem with cluster, which I cannot manage to resolved.
There is long thread about the problem with failed defaultGuestNetwork

Now:
What I am going to do:
* download all VM volumes
* reinstall Cloudstack from scratch
* upload volumes to new VMs.

How can that be done correctly?
In other words - what is correct way to migrate all VMs? Is there an
official guide for that?

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

Under this defaultGuestNetwork, I go to Virtual Appliances. There is no VMS
- "no data to show"

I dont have any network, other than this single default one.

I've tried adding new network - Add guest network. But I am not able to do
so, cause in the wizard popup, it offers empty dropdown with Zones
selection. And this wizard doesnt not allow to go further without selecting
Zone

On Wed, Mar 20, 2019 at 10:28 AM Andrija Panic 
wrote:

> you need to delete/remove all VMs inside this network (tick the "Expunge"
> button during VM deletion - if you want to really delete the VMs) in order
> to be able to delete the network - OR simply attach this VM to another
> network, make this new network a DEFAULT one (NIC that is...), and then
> detach from old network - and then effectively your VM was "removed" from
> old network - after this you should be able to delete the old network. I
> assume some DB incosistencies perhaps, being the reason you can not restart
> the network.
>
> Did you try restarting some other Network - or deploying a new network,
> spin a VM in it, then again try to restart this new network - does it work
> ?
>
> Andrija
>
> On Wed, 20 Mar 2019 at 08:58, Jevgeni Zolotarjov 
> wrote:
>
> > >>>Stop mgmt,
> > >>>Stop all agents
> > >>>Restart libvirtd (and check libvirt logs afterwards)
> > >>>Start agents
> > >>>Start mgmt.
> >
> > I did that numerous time. Nothing really suspicious
> > I can see that systems VMs are running - both in cloudstack console and
> > with virsh list -all
> >
> > It is apparently problem with network.
> > Is there a way to force recreation of defaultGuestNetwork? or force
> > recreation of Virtual Router.
> > I am unable to delete network, which is supposed to rebuild network with
> > its router. Thats the issue
> >
> > The issue with libvirtd was, that eventually at some point it was updated
> > during 4 months of running, and not rebooted. It still worked. We had to
> > add listen_tcp = 1 for libvirtd to start working again.
> >
> > On Wed, Mar 20, 2019 at 9:49 AM Andrija Panic 
> > wrote:
> >
> > > As Sergey suggested... but i would also verify no libvirt issues or
> > storage
> > > pool issues - so perhaps:
> > >
> > > Stop mgmt,
> > > Stop all agents
> > > Restart libvirtd (and check libvirt logs afterwards)
> > > Start agents
> > > Start mgmt.
> > >
> > > What was originally issue with libvirtd ?
> > > That sounds fishy to me...
> > >
> > > Andrija
> > >
> > > On Wed, Mar 20, 2019, 02:15 Sergey Levitskiy 
> > wrote:
> > >
> > > > select * from networks where removed is null;
> > > > select * from vm_instance where id=87;
> > > > select id,name from vm_instance where name like 'r%' and removed is
> > null;
> > > >
> > > > Basically since the network offering is not redundant this error is
> > only
> > > > thrown when there is no router associated with your network. Usually
> > > > management server restart tries to implement network again. Please
> > > restart
> > > > management server, save and share management server log.
> > > >
> > > >
> > > >
> > > >
> > > > On 3/19/19, 3:31 PM, "Jevgeni Zolotarjov" 
> > > wrote:
> > > >
> > > > >>>>Check network_offering table for  value in column
> > > > redundant_router_service  for the network offering you use.
> > > > in table network_offering_table all records have
> > > > redundant_router_service =
> > > > 0
> > > >
> > > > Can you also run the following:
> > > > >>>select name, state, removed  from host where name like 'r%'
> > > > returns zero rows - nothing
> > > >
> > > > >>>select * from domain_router;
> > > > # id, element_id, public_mac_address, public_ip_address,
> > > > public_netmask,
> > > > guest_netmask, guest_ip_address, is_redundant_router, priority,
> > > > redundant_state, stop_pending, role, template_version,
> > > scripts_version,
> > > > vpc_id, update_state
> > > > '4', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN',
> '0',
> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28
> > UTC
> > > > 2018',
> > > &g

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

>>>Stop mgmt,
>>>Stop all agents
>>>Restart libvirtd (and check libvirt logs afterwards)
>>>Start agents
>>>Start mgmt.

I did that numerous time. Nothing really suspicious
I can see that systems VMs are running - both in cloudstack console and
with virsh list -all

It is apparently problem with network.
Is there a way to force recreation of defaultGuestNetwork? or force
recreation of Virtual Router.
I am unable to delete network, which is supposed to rebuild network with
its router. Thats the issue

The issue with libvirtd was, that eventually at some point it was updated
during 4 months of running, and not rebooted. It still worked. We had to
add listen_tcp = 1 for libvirtd to start working again.

On Wed, Mar 20, 2019 at 9:49 AM Andrija Panic 
wrote:

> As Sergey suggested... but i would also verify no libvirt issues or storage
> pool issues - so perhaps:
>
> Stop mgmt,
> Stop all agents
> Restart libvirtd (and check libvirt logs afterwards)
> Start agents
> Start mgmt.
>
> What was originally issue with libvirtd ?
> That sounds fishy to me...
>
> Andrija
>
> On Wed, Mar 20, 2019, 02:15 Sergey Levitskiy  wrote:
>
> > select * from networks where removed is null;
> > select * from vm_instance where id=87;
> > select id,name from vm_instance where name like 'r%' and removed is null;
> >
> > Basically since the network offering is not redundant this error is only
> > thrown when there is no router associated with your network. Usually
> > management server restart tries to implement network again. Please
> restart
> > management server, save and share management server log.
> >
> >
> >
> >
> > On 3/19/19, 3:31 PM, "Jevgeni Zolotarjov" 
> wrote:
> >
> > >>>>Check network_offering table for  value in column
> > redundant_router_service  for the network offering you use.
> > in table network_offering_table all records have
> > redundant_router_service =
> > 0
> >
> > Can you also run the following:
> > >>>select name, state, removed  from host where name like 'r%'
> > returns zero rows - nothing
> >
> > >>>select * from domain_router;
> > # id, element_id, public_mac_address, public_ip_address,
> > public_netmask,
> > guest_netmask, guest_ip_address, is_redundant_router, priority,
> > redundant_state, stop_pending, role, template_version,
> scripts_version,
> > vpc_id, update_state
> > '4', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
> > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC
> > 2018',
> > '57db7bd8118977a5f2cd3ef1c7503633\n', NULL, NULL
> > '49', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
> > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC
> > 2018',
> > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > '73', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
> > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC
> > 2018',
> > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > '74', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
> > 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL
> > '75', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
> > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC
> > 2018',
> > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > '76', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
> > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC
> > 2018',
> > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > '77', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
> > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC
> > 2018',
> > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, 'UPDATE_FAILED'
> > '80', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
> > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC
> > 2018',
> > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > '85', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
> > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC
> > 2018',
> > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > '86', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
> > 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL
> > '87', '1', NULL, NULL, NULL, NULL, NULL, '0', N

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

>>>>Check network_offering table for  value in column
redundant_router_service  for the network offering you use.
in table network_offering_table all records have redundant_router_service =
0

Can you also run the following:
>>>select name, state, removed  from host where name like 'r%'
returns zero rows - nothing

>>>select * from domain_router;
# id, element_id, public_mac_address, public_ip_address, public_netmask,
guest_netmask, guest_ip_address, is_redundant_router, priority,
redundant_state, stop_pending, role, template_version, scripts_version,
vpc_id, update_state
'4', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018',
'57db7bd8118977a5f2cd3ef1c7503633\n', NULL, NULL
'49', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018',
'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
'73', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018',
'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
'74', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL
'75', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018',
'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
'76', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018',
'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
'77', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC 2018',
'c03a474302d89fa82d345e10fe4cb751\n', NULL, 'UPDATE_FAILED'
'80', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC 2018',
'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
'85', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC 2018',
'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
'86', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL
'87', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0',
'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.2.0 Mon Nov 12 15:06:49 UTC
2018', '873057731ff2cba4a1f3b2411765407c\n', NULL, NULL


>>>select * from router_network_ref;
# id, router_id, network_id, guest_type
'1', '4', '204', 'Shared'
'2', '49', '204', 'Shared'
'3', '73', '204', 'Shared'
'4', '75', '204', 'Shared'
'5', '76', '204', 'Shared'
'6', '77', '204', 'Shared'
'7', '80', '204', 'Shared'
'8', '85', '204', 'Shared'
'9', '86', '204', 'Shared'
'10', '87', '204', 'Shared'


On Wed, Mar 20, 2019 at 12:18 AM Sergey Levitskiy 
wrote:

> Check network_offering table for  value in column
> redundant_router_service  for the network offering you use.
> Can you also run the following:
> select name, state, removed  from host where name like 'r%'
> select * from domain_router;
> select * from router_network_ref;
>
> Cloudstack is supposed to recreate you VR. If it is not happening there is
> something fundamentally wrong. I would advise to destroy your VR again.
> Stop you management server. Rotate management server log and start it
> again. If your VR doesn't start in few min, post your complete  management
> server log  and agent log again.
>
>
>
>
> On 3/19/19, 2:56 PM, "Jevgeni Zolotarjov"  wrote:
>
> >>>Network offering needs to be change to non-redundant
> How do I do that?
>
> On Tue, Mar 19, 2019 at 11:47 PM Sergey Levitskiy  >
> wrote:
>
> > Network offering needs to be change to non-redundant. Most likely
> old bug
> > is resurfaced.
> > https://issues.apache.org/jira/browse/CLOUDSTACK-9024
> >
> >
> >
> > On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov" 
> wrote:
> >
> > Hello
> >
> > I did exactly like you suggested.
> >
> > After UPDATE on db, I can see the router in cloudstack console.
> > But attempt to restart network fails.
> > I get error:
> > Resource [DataCenter:1] is unreachable: Can't find all necessary
> > running
> > routers!
> >
> > I rechecked agents on both servers. They look running OK.
> >
> > On Tue, Mar 19, 2019 at 10:00 PM Andrija Panic <
> > andrija.pa...@gmail.com>
> > wrote:
> >
> > > Ok, so:
> > >
> > > 

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

>>>Network offering needs to be change to non-redundant
How do I do that?

On Tue, Mar 19, 2019 at 11:47 PM Sergey Levitskiy 
wrote:

> Network offering needs to be change to non-redundant. Most likely old bug
> is resurfaced.
> https://issues.apache.org/jira/browse/CLOUDSTACK-9024
>
>
>
> On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov"  wrote:
>
> Hello
>
> I did exactly like you suggested.
>
> After UPDATE on db, I can see the router in cloudstack console.
> But attempt to restart network fails.
> I get error:
> Resource [DataCenter:1] is unreachable: Can't find all necessary
> running
> routers!
>
> I rechecked agents on both servers. They look running OK.
>
> On Tue, Mar 19, 2019 at 10:00 PM Andrija Panic <
> andrija.pa...@gmail.com>
> wrote:
>
> > Ok, so:
> >
> > 1.BACKUP YOUR DB - in case there is issues, you will want to restore
> > 2. I tried to reproduce your case on 4.11.2 (though didn't do any
> > maintenance etc.) - and could not - I have artificially marked
> existing VR
> > as removed in DB - and tried to restart network and it worked just
> fine.
> >
> > Let's try to "undelete" the VR for that network - though I can't be
> sure if
> > this will work or not.
> >
> > Find VR based on the network UUID:
> >   SELECT * FROM vm_instance WHERE id IN (SELECT instance_id from
> nics
> > where network_id IN (SELECT id FROM networks WHERE
> > UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73"));
> > Obviously, replace this UUID with your network UUID.
> > SQL above will return all VMs (user VMs, VRs, etc, delete or not...)
> that
> > have NIC inside the network with that UUID - look for the r-XX-VM
> being a
> > router.
> >
> > Now that you have the name of the VR (i.e. r-4-VM) - you want to set
> 2
> > fields to some other value than what it might be already, as below:
> >
> > Need to set field "state" to Stopped:
> >   UPDATE cloud.vm_instance SET state='Stopped' WHERE
> name="r-4-VM";
> >  (what state is set currently ???)
> >
> > Need to set "removed" to NULL value:
> >   UPDATE cloud.vm_instance SET removed = NULL WHERE
> NAME="r-4-VM"; (is
> > there a removed date already set ???)
> >
> > Obviously make sure that this VR is not running on any hypervisor
> and if
> > running, virsh destroy it...
> >
> > AFTER above has been done (VR is considered to exist but stopped,
> from
> > CloudStack point of view) - try to RESTART the network (don't bother
> > deleteing it, since there are VMs in that network)
> >
> > I'm not 100% positive this will fix your issue, but doesn't hurt to
> try
> >
> > If above doesn't work - I would still take a look into agents and if
> they
> > are still connected - optionally, restart agents on both hosts once
> more
> > and confirm they are connected and up.
> >
> > Let us know how it goes (and please backup DB once more before any
> actions
> > !)
> >
>     > Andrija
> >
> >
> > On Tue, 19 Mar 2019 at 20:41, Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> > wrote:
> >
> > > Yes.  Just a single network.
> > >
> > > On Tue, 19 Mar 2019, 21:39 Andrija Panic,  >
> > wrote:
> > >
> > > > Just one more clarification - this is Isolate single network (not
> > Shared
> > > > Network, not VPC) ?
> > > >
> > > >
> > > >
> > > > On Tue, Mar 19, 2019, 19:36 Jevgeni Zolotarjov <
> j.zolotar...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Name defaultGuestNetwork
> > > > > ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1
> > > > > Zone PBT zone 1
> > > > > Description defaultGuestNetwork
> > > > > Type Shared
> > > > > State Setup
> > > > > VPC ID N/A
> > > > > Persistent No
> > > > >
> > > > > On Tue, Mar 19, 2019 at 8:29 PM Andrija Panic <
> > andrija.pa...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Share the network id and n

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

Hello

I did exactly like you suggested.

After UPDATE on db, I can see the router in cloudstack console.
But attempt to restart network fails.
I get error:
Resource [DataCenter:1] is unreachable: Can't find all necessary running
routers!

I rechecked agents on both servers. They look running OK.

On Tue, Mar 19, 2019 at 10:00 PM Andrija Panic 
wrote:

> Ok, so:
>
> 1.BACKUP YOUR DB - in case there is issues, you will want to restore
> 2. I tried to reproduce your case on 4.11.2 (though didn't do any
> maintenance etc.) - and could not - I have artificially marked existing VR
> as removed in DB - and tried to restart network and it worked just fine.
>
> Let's try to "undelete" the VR for that network - though I can't be sure if
> this will work or not.
>
> Find VR based on the network UUID:
>   SELECT * FROM vm_instance WHERE id IN (SELECT instance_id from nics
> where network_id IN (SELECT id FROM networks WHERE
> UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73"));
> Obviously, replace this UUID with your network UUID.
> SQL above will return all VMs (user VMs, VRs, etc, delete or not...) that
> have NIC inside the network with that UUID - look for the r-XX-VM being a
> router.
>
> Now that you have the name of the VR (i.e. r-4-VM) - you want to set 2
> fields to some other value than what it might be already, as below:
>
> Need to set field "state" to Stopped:
>   UPDATE cloud.vm_instance SET state='Stopped' WHERE name="r-4-VM";
>  (what state is set currently ???)
>
> Need to set "removed" to NULL value:
>   UPDATE cloud.vm_instance SET removed = NULL WHERE NAME="r-4-VM"; (is
> there a removed date already set ???)
>
> Obviously make sure that this VR is not running on any hypervisor and if
> running, virsh destroy it...
>
> AFTER above has been done (VR is considered to exist but stopped, from
> CloudStack point of view) - try to RESTART the network (don't bother
> deleteing it, since there are VMs in that network)
>
> I'm not 100% positive this will fix your issue, but doesn't hurt to try
>
> If above doesn't work - I would still take a look into agents and if they
> are still connected - optionally, restart agents on both hosts once more
> and confirm they are connected and up.
>
> Let us know how it goes (and please backup DB once more before any actions
> !)
>
> Andrija
>
>
> On Tue, 19 Mar 2019 at 20:41, Jevgeni Zolotarjov 
> wrote:
>
> > Yes.  Just a single network.
> >
> > On Tue, 19 Mar 2019, 21:39 Andrija Panic, 
> wrote:
> >
> > > Just one more clarification - this is Isolate single network (not
> Shared
> > > Network, not VPC) ?
> > >
> > >
> > >
> > > On Tue, Mar 19, 2019, 19:36 Jevgeni Zolotarjov  >
> > > wrote:
> > >
> > > > Name defaultGuestNetwork
> > > > ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1
> > > > Zone PBT zone 1
> > > > Description defaultGuestNetwork
> > > > Type Shared
> > > > State Setup
> > > > VPC ID N/A
> > > > Persistent No
> > > >
> > > > On Tue, Mar 19, 2019 at 8:29 PM Andrija Panic <
> andrija.pa...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Share the network id and name as seen from GUI...
> > > > >
> > > > > On Tue, Mar 19, 2019, 19:27 Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > >>> 1. Confirm please rolling restart is set to false please -
> > double
> > > > > check
> > > > > > Double checked - It is set to false
> > > > > >
> > > > > > >>>>2. If so - do you know the name of VR which you deleted ? Is
> it
> > > > last
> > > > > > one
> > > > > > >>> ever created - if so we can find it easily...
> > > > > > I dont know the name
> > > > > > Is there a way to fetch it from DB?
> > > > > >
> > > > > > On Tue, Mar 19, 2019 at 7:58 PM Andrija Panic <
> > > andrija.pa...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Right...still complaining on missing running routers.
> > > > > > >
> > > > > > > 1. Confirm please rolling restart is set to false please -
> double
> > > > check
> > > > > > > 2. If 

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

Yes.  Just a single network.

On Tue, 19 Mar 2019, 21:39 Andrija Panic,  wrote:

> Just one more clarification - this is Isolate single network (not Shared
> Network, not VPC) ?
>
>
>
> On Tue, Mar 19, 2019, 19:36 Jevgeni Zolotarjov 
> wrote:
>
> > Name defaultGuestNetwork
> > ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1
> > Zone PBT zone 1
> > Description defaultGuestNetwork
> > Type Shared
> > State Setup
> > VPC ID N/A
> > Persistent No
> >
> > On Tue, Mar 19, 2019 at 8:29 PM Andrija Panic 
> > wrote:
> >
> > > Share the network id and name as seen from GUI...
> > >
> > > On Tue, Mar 19, 2019, 19:27 Jevgeni Zolotarjov  >
> > > wrote:
> > >
> > > > >>> 1. Confirm please rolling restart is set to false please - double
> > > check
> > > > Double checked - It is set to false
> > > >
> > > > >>>>2. If so - do you know the name of VR which you deleted ? Is it
> > last
> > > > one
> > > > >>> ever created - if so we can find it easily...
> > > > I dont know the name
> > > > Is there a way to fetch it from DB?
> > > >
> > > > On Tue, Mar 19, 2019 at 7:58 PM Andrija Panic <
> andrija.pa...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Right...still complaining on missing running routers.
> > > > >
> > > > > 1. Confirm please rolling restart is set to false please - double
> > check
> > > > > 2. If so - do you know the name of VR which you deleted ? Is it
> last
> > > one
> > > > > ever created - if so we can find it easily...
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Mar 19, 2019, 18:40 Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Here is management server log
> > > > > >
> https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 19, 2019 at 7:29 PM Andrija Panic <
> > > andrija.pa...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I would also try to "undelete" VR in DB, but let's keep this as
> > > last
> > > > > > step.
> > > > > > >
> > > > > > > On Tue, Mar 19, 2019, 18:24 Andrija Panic <
> > andrija.pa...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Disclaimer: from mobile device.
> > > > > > > >
> > > > > > > > I don't see any special failure in agent log, except some
> long
> > > > > running
> > > > > > > > migration, timeout for graceful VM shutdown etc (and agent
> > > restart)
> > > > > > > >
> > > > > > > > You did not send mgmt log after last network restart failure
> ?
> > > > > > > >
> > > > > > > > On Tue, Mar 19, 2019, 17:29 Jevgeni Zolotarjov <
> > > > > j.zolotar...@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Under
> > > > > > > >> Infrastructure / Hosts:
> > > > > > > >> both hosts are enabled and green (Unsecure though)
> > > > > > > >>
> > > > > > > >> agent.log -
> > > > > > > >>
> > > > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
> > > > > > > >>
> > > > > > > >> I set network.rolling.restart to false now. Restarted
> > management
> > > > > > server
> > > > > > > -
> > > > > > > >> same problem - cannot restart not delete network
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Tue, Mar 19, 2019 at 5:48 PM Boris Stoyanov <
> > > > > > > >> boris.stoya...@shapeblue.com>
> > > > > > > >> wrote:
> > > > > > > >>

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

Name defaultGuestNetwork
ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1
Zone PBT zone 1
Description defaultGuestNetwork
Type Shared
State Setup
VPC ID N/A
Persistent No

On Tue, Mar 19, 2019 at 8:29 PM Andrija Panic 
wrote:

> Share the network id and name as seen from GUI...
>
> On Tue, Mar 19, 2019, 19:27 Jevgeni Zolotarjov 
> wrote:
>
> > >>> 1. Confirm please rolling restart is set to false please - double
> check
> > Double checked - It is set to false
> >
> > >>>>2. If so - do you know the name of VR which you deleted ? Is it last
> > one
> > >>> ever created - if so we can find it easily...
> > I dont know the name
> > Is there a way to fetch it from DB?
> >
> > On Tue, Mar 19, 2019 at 7:58 PM Andrija Panic 
> > wrote:
> >
> > > Right...still complaining on missing running routers.
> > >
> > > 1. Confirm please rolling restart is set to false please - double check
> > > 2. If so - do you know the name of VR which you deleted ? Is it last
> one
> > > ever created - if so we can find it easily...
> > >
> > >
> > >
> > > On Tue, Mar 19, 2019, 18:40 Jevgeni Zolotarjov  >
> > > wrote:
> > >
> > > > Here is management server log
> > > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> > > >
> > > >
> > > > On Tue, Mar 19, 2019 at 7:29 PM Andrija Panic <
> andrija.pa...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > I would also try to "undelete" VR in DB, but let's keep this as
> last
> > > > step.
> > > > >
> > > > > On Tue, Mar 19, 2019, 18:24 Andrija Panic  >
> > > > wrote:
> > > > >
> > > > > > Disclaimer: from mobile device.
> > > > > >
> > > > > > I don't see any special failure in agent log, except some long
> > > running
> > > > > > migration, timeout for graceful VM shutdown etc (and agent
> restart)
> > > > > >
> > > > > > You did not send mgmt log after last network restart failure ?
> > > > > >
> > > > > > On Tue, Mar 19, 2019, 17:29 Jevgeni Zolotarjov <
> > > j.zolotar...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Under
> > > > > >> Infrastructure / Hosts:
> > > > > >> both hosts are enabled and green (Unsecure though)
> > > > > >>
> > > > > >> agent.log -
> > > > > >>
> > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
> > > > > >>
> > > > > >> I set network.rolling.restart to false now. Restarted management
> > > > server
> > > > > -
> > > > > >> same problem - cannot restart not delete network
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Mar 19, 2019 at 5:48 PM Boris Stoyanov <
> > > > > >> boris.stoya...@shapeblue.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi, you shouldn’t be worried about data as long as you use a
> > > > separate
> > > > > >> > shared storage.
> > > > > >> >
> > > > > >> > What does the cloudstack console say about your host? Is it
> up?
> > If
> > > > > it’s
> > > > > >> up
> > > > > >> > then you should be able to deploy a VM on it.
> > > > > >> >
> > > > > >> > If not, you’ll need to investigate the reason cloudstack-agent
> > is
> > > > not
> > > > > >> > connected, is the agent running? If not just start it and see
> if
> > > it
> > > > > >> > connects.
> > > > > >> >
> > > > > >> > If it’s running but still not connected please share the
> > > agent.log.
> > > > > >> >
> > > > > >> > If nothing helps at all you can remove the host and add it
> > again,
> > > > but
> > > > > >> > let’s try to troubleshoot first.
> > > > > >> >
> > > > > >> > Bobby.
> > > > > >> >
> &g

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

>>> 1. Confirm please rolling restart is set to false please - double check
Double checked - It is set to false

>>>>2. If so - do you know the name of VR which you deleted ? Is it last one
>>> ever created - if so we can find it easily...
I dont know the name
Is there a way to fetch it from DB?

On Tue, Mar 19, 2019 at 7:58 PM Andrija Panic 
wrote:

> Right...still complaining on missing running routers.
>
> 1. Confirm please rolling restart is set to false please - double check
> 2. If so - do you know the name of VR which you deleted ? Is it last one
> ever created - if so we can find it easily...
>
>
>
> On Tue, Mar 19, 2019, 18:40 Jevgeni Zolotarjov 
> wrote:
>
> > Here is management server log
> > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> >
> >
> > On Tue, Mar 19, 2019 at 7:29 PM Andrija Panic 
> > wrote:
> >
> > > I would also try to "undelete" VR in DB, but let's keep this as last
> > step.
> > >
> > > On Tue, Mar 19, 2019, 18:24 Andrija Panic 
> > wrote:
> > >
> > > > Disclaimer: from mobile device.
> > > >
> > > > I don't see any special failure in agent log, except some long
> running
> > > > migration, timeout for graceful VM shutdown etc (and agent restart)
> > > >
> > > > You did not send mgmt log after last network restart failure ?
> > > >
> > > > On Tue, Mar 19, 2019, 17:29 Jevgeni Zolotarjov <
> j.zolotar...@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> Under
> > > >> Infrastructure / Hosts:
> > > >> both hosts are enabled and green (Unsecure though)
> > > >>
> > > >> agent.log -
> > > >> https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
> > > >>
> > > >> I set network.rolling.restart to false now. Restarted management
> > server
> > > -
> > > >> same problem - cannot restart not delete network
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Mar 19, 2019 at 5:48 PM Boris Stoyanov <
> > > >> boris.stoya...@shapeblue.com>
> > > >> wrote:
> > > >>
> > > >> > Hi, you shouldn’t be worried about data as long as you use a
> > separate
> > > >> > shared storage.
> > > >> >
> > > >> > What does the cloudstack console say about your host? Is it up? If
> > > it’s
> > > >> up
> > > >> > then you should be able to deploy a VM on it.
> > > >> >
> > > >> > If not, you’ll need to investigate the reason cloudstack-agent is
> > not
> > > >> > connected, is the agent running? If not just start it and see if
> it
> > > >> > connects.
> > > >> >
> > > >> > If it’s running but still not connected please share the
> agent.log.
> > > >> >
> > > >> > If nothing helps at all you can remove the host and add it again,
> > but
> > > >> > let’s try to troubleshoot first.
> > > >> >
> > > >> > Bobby.
> > > >> >
> > > >> >
> > > >> > boris.stoya...@shapeblue.com
> > > >> > www.shapeblue.com
> > > >> > Amadeus House, Floral Street, London  WC2E 9DPUK
> > > >> > @shapeblue
> > > >> >
> > > >> >
> > > >> >
> > > >> > > On 19 Mar 2019, at 17:29, Jevgeni Zolotarjov <
> > > j.zolotar...@gmail.com>
> > > >> > wrote:
> > > >> > >
> > > >> > > Guys, please help with it.
> > > >> > > What can be done here?
> > > >> > > There is too much valuable data.
> > > >> > >
> > > >> > > On Tue, Mar 19, 2019 at 4:21 PM Jevgeni Zolotarjov <
> > > >> > j.zolotar...@gmail.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > >> Tried that just now and got error:
> > > >> > >> Resource [DataCenter:1] is unreachable: Can't find all
> necessary
> > > >> running
> > > >> > >> routers!
> > > >> > >>
> > 

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

Here is management server log
https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A


On Tue, Mar 19, 2019 at 7:29 PM Andrija Panic 
wrote:

> I would also try to "undelete" VR in DB, but let's keep this as last step.
>
> On Tue, Mar 19, 2019, 18:24 Andrija Panic  wrote:
>
> > Disclaimer: from mobile device.
> >
> > I don't see any special failure in agent log, except some long running
> > migration, timeout for graceful VM shutdown etc (and agent restart)
> >
> > You did not send mgmt log after last network restart failure ?
> >
> > On Tue, Mar 19, 2019, 17:29 Jevgeni Zolotarjov 
> > wrote:
> >
> >> Under
> >> Infrastructure / Hosts:
> >> both hosts are enabled and green (Unsecure though)
> >>
> >> agent.log -
> >> https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
> >>
> >> I set network.rolling.restart to false now. Restarted management server
> -
> >> same problem - cannot restart not delete network
> >>
> >>
> >>
> >> On Tue, Mar 19, 2019 at 5:48 PM Boris Stoyanov <
> >> boris.stoya...@shapeblue.com>
> >> wrote:
> >>
> >> > Hi, you shouldn’t be worried about data as long as you use a separate
> >> > shared storage.
> >> >
> >> > What does the cloudstack console say about your host? Is it up? If
> it’s
> >> up
> >> > then you should be able to deploy a VM on it.
> >> >
> >> > If not, you’ll need to investigate the reason cloudstack-agent is not
> >> > connected, is the agent running? If not just start it and see if it
> >> > connects.
> >> >
> >> > If it’s running but still not connected please share the agent.log.
> >> >
> >> > If nothing helps at all you can remove the host and add it again, but
> >> > let’s try to troubleshoot first.
> >> >
> >> > Bobby.
> >> >
> >> >
> >> > boris.stoya...@shapeblue.com
> >> > www.shapeblue.com
> >> > Amadeus House, Floral Street, London  WC2E 9DPUK
> >> > @shapeblue
> >> >
> >> >
> >> >
> >> > > On 19 Mar 2019, at 17:29, Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> >> > wrote:
> >> > >
> >> > > Guys, please help with it.
> >> > > What can be done here?
> >> > > There is too much valuable data.
> >> > >
> >> > > On Tue, Mar 19, 2019 at 4:21 PM Jevgeni Zolotarjov <
> >> > j.zolotar...@gmail.com>
> >> > > wrote:
> >> > >
> >> > >> Tried that just now and got error:
> >> > >> Resource [DataCenter:1] is unreachable: Can't find all necessary
> >> running
> >> > >> routers!
> >> > >>
> >> > >> In the log I see:
> >> > >> =
> >> > >>
> >> > >> 2019-03-19 14:20:39,644 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648)
> >> (logid:265a6099)
> >> > >> Restarting network 204...
> >> > >> 2019-03-19 14:20:39,645 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648)
> >> (logid:265a6099)
> >> > >> Performing rolling restart of routers of network Ntwk[204|Guest|6]
> >> > >> 2019-03-19 14:20:39,649 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648)
> >> (logid:265a6099)
> >> > >> Asking VirtualRouter to implemenet Ntwk[204|Guest|6]
> >> > >> 2019-03-19 14:20:39,658 ERROR [c.c.a.ApiAsyncJobDispatcher]
> >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099)
> >> Unexpected
> >> > >> exception while executing
> >> > >> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
> >> > >> com.cloud.exception.ResourceUnavailableException: Resource
> >> > [DataCenter:1]
> >> > >> is unreachable: Can't find all necessary running routers!
> >> > >>at
> >> > >>
> >> >
> >>
> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouter

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

Under
Infrastructure / Hosts:
both hosts are enabled and green (Unsecure though)

agent.log -
https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-

I set network.rolling.restart to false now. Restarted management server -
same problem - cannot restart not delete network



On Tue, Mar 19, 2019 at 5:48 PM Boris Stoyanov 
wrote:

> Hi, you shouldn’t be worried about data as long as you use a separate
> shared storage.
>
> What does the cloudstack console say about your host? Is it up? If it’s up
> then you should be able to deploy a VM on it.
>
> If not, you’ll need to investigate the reason cloudstack-agent is not
> connected, is the agent running? If not just start it and see if it
> connects.
>
> If it’s running but still not connected please share the agent.log.
>
> If nothing helps at all you can remove the host and add it again, but
> let’s try to troubleshoot first.
>
> Bobby.
>
>
> boris.stoya...@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London  WC2E 9DPUK
> @shapeblue
>
>
>
> > On 19 Mar 2019, at 17:29, Jevgeni Zolotarjov 
> wrote:
> >
> > Guys, please help with it.
> > What can be done here?
> > There is too much valuable data.
> >
> > On Tue, Mar 19, 2019 at 4:21 PM Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> > wrote:
> >
> >> Tried that just now and got error:
> >> Resource [DataCenter:1] is unreachable: Can't find all necessary running
> >> routers!
> >>
> >> In the log I see:
> >> =
> >>
> >> 2019-03-19 14:20:39,644 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648) (logid:265a6099)
> >> Restarting network 204...
> >> 2019-03-19 14:20:39,645 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648) (logid:265a6099)
> >> Performing rolling restart of routers of network Ntwk[204|Guest|6]
> >> 2019-03-19 14:20:39,649 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648) (logid:265a6099)
> >> Asking VirtualRouter to implemenet Ntwk[204|Guest|6]
> >> 2019-03-19 14:20:39,658 ERROR [c.c.a.ApiAsyncJobDispatcher]
> >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) Unexpected
> >> exception while executing
> >> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
> >> com.cloud.exception.ResourceUnavailableException: Resource
> [DataCenter:1]
> >> is unreachable: Can't find all necessary running routers!
> >>at
> >>
> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243)
> >>at
> >>
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203)
> >>at
> >>
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948)
> >>at
> >>
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853)
> >>at
> >>
> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
> >>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> >>at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> Source)
> >>at java.lang.reflect.Method.invoke(Unknown Source)
> >>at
> >>
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> >>at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> >>at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >>at
> >>
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> >>at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> >>at
> >>
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> >>at
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> >>at
> >>
> org.springframework.aop.i

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

Guys, please help with it.
What can be done here?
There is too much valuable data.

On Tue, Mar 19, 2019 at 4:21 PM Jevgeni Zolotarjov 
wrote:

> Tried that just now and got error:
> Resource [DataCenter:1] is unreachable: Can't find all necessary running
> routers!
>
> In the log I see:
> =
>
> 2019-03-19 14:20:39,644 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648) (logid:265a6099)
> Restarting network 204...
> 2019-03-19 14:20:39,645 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648) (logid:265a6099)
> Performing rolling restart of routers of network Ntwk[204|Guest|6]
> 2019-03-19 14:20:39,649 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648) (logid:265a6099)
> Asking VirtualRouter to implemenet Ntwk[204|Guest|6]
> 2019-03-19 14:20:39,658 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) Unexpected
> exception while executing
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
> com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1]
> is unreachable: Can't find all necessary running routers!
> at
> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853)
> at
> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> at com.sun.proxy.$Proxy229.restartNetwork(Unknown Source)
> at
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at java.util.concurrent.

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

due to "Can't delete the network, not all
> user vms are expunged. Vm
> VM[User|i-2-11-VM] is in Stopped state" - which is fine.
>
> You should be able to just start the user VM - but if you have actually
> delete the VR itself, then just do Network restart with "cleanup" and it
> will recreate a new VR, after which you should be able to start the VM.
>
> Andrija
>
> andrija.pa...@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London  WC2E 9DPUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: Jevgeni Zolotarjov 
> Sent: 19 March 2019 15:10
> To: users@cloudstack.apache.org
> Subject: Re: Disaster after maintenance
>
> I mean I cannot delete network: In the management server log I see
>
> ==
> 019-03-19 14:06:36,316 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Executing
> AsyncJobVO {id:5090, userId: 2, accountId: 2, instanceType: Network,
> instanceId: 204, cmd:
> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
>
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
>
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 264216221068220, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: null}
> 2019-03-19 14:06:36,351 WARN  [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-1:ctx-1c0fd4dc job-5090 ctx-134954fa) (logid:c734edfc)
> Can't delete the network, not all user vms are expunged. Vm
> VM[User|i-2-11-VM] is in Stopped state
> 2019-03-19 14:06:36,356 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Complete async
> job-5090, jobStatus: FAILED, resultCode: 530, result:
>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> to delete network"}
> ==
>
>
> I deleted a router, expecting it to be recreated on deleting network. But
> I am unable to delete network because of above error
>
> On Tue, Mar 19, 2019 at 3:58 PM Jevgeni Zolotarjov  >
> wrote:
>
> > I've managed to make libvirtd running
> > Now cloudstack console shows both hosts - running
> >
> > But now as I have removed network, VMs are unable to start.
> >
> > How can I recreate the network now?
> >
> > On Tue, Mar 19, 2019 at 3:14 PM Ivan Kudryavtsev
> > 
> > wrote:
> >
> >> Jevgeniy, it may be a documentation bug. Take s look:
> >> https://github.com/apache/cloudstack-documentation/pull/27/files
> >>
> >> вт, 19 мар. 2019 г., 9:09 Jevgeni Zolotarjov :
> >>
> >> > That's it - libvirtd failed to start on second host.
> >> > Tried restarting, but it does not start.
> >> >
> >> >
> >> > >> Do you have some NUMA constraints or anything which requires
> >> particular
> >> > RAM configuration?
> >> > No
> >> >
> >> >  libvirtd.service - Virtualization daemon
> >> >Loaded: loaded (/usr/lib/systemd/system/libvirtd.service;
> >> > enabled; vendor preset: enabled)
> >> >Active: failed (Result: start-limit) since Tue 2019-03-19
> >> > 13:03:07
> >> GMT;
> >> > 12s ago
> >> >  Docs: man:libvirtd(8)
> >> >https://libvirt.org
> >> >   Process: 892 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS
> >> > (code=exited,
> >> > status=1/FAILURE)
> >> >  Main PID: 892 (code=exited, status=1/FAILURE)
> >> > Tasks: 19 (limit: 32768)
> >> >CGroup: /system.slice/libvirtd.service
> >> >├─11338 /usr/sbin/libvirtd -d -l
> >> >├─11909 /usr/sbin/dnsmasq
> >> > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
> >> > --dhcp-script=/usr/libexec/libvirt_leaseshelper
> >> >└─11910 /usr/sbin/dnsmasq
> >> > --conf-file

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

I mean I cannot delete network: In the management server log I see

==
019-03-19 14:06:36,316 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Executing
AsyncJobVO {id:5090, userId: 2, accountId: 2, instanceType: Network,
instanceId: 204, cmd:
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
{"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 264216221068220, completeMsid: null, lastUpdated:
null, lastPolled: null, created: null}
2019-03-19 14:06:36,351 WARN  [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-1:ctx-1c0fd4dc job-5090 ctx-134954fa) (logid:c734edfc)
Can't delete the network, not all user vms are expunged. Vm
VM[User|i-2-11-VM] is in Stopped state
2019-03-19 14:06:36,356 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Complete async
job-5090, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to delete network"}
==


I deleted a router, expecting it to be recreated on deleting network. But I
am unable to delete network because of above error

On Tue, Mar 19, 2019 at 3:58 PM Jevgeni Zolotarjov 
wrote:

> I've managed to make libvirtd running
> Now cloudstack console shows both hosts - running
>
> But now as I have removed network, VMs are unable to start.
>
> How can I recreate the network now?
>
> On Tue, Mar 19, 2019 at 3:14 PM Ivan Kudryavtsev 
> wrote:
>
>> Jevgeniy, it may be a documentation bug. Take s look:
>> https://github.com/apache/cloudstack-documentation/pull/27/files
>>
>> вт, 19 мар. 2019 г., 9:09 Jevgeni Zolotarjov :
>>
>> > That's it - libvirtd failed to start on second host.
>> > Tried restarting, but it does not start.
>> >
>> >
>> > >> Do you have some NUMA constraints or anything which requires
>> particular
>> > RAM configuration?
>> > No
>> >
>> >  libvirtd.service - Virtualization daemon
>> >Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
>> > vendor preset: enabled)
>> >Active: failed (Result: start-limit) since Tue 2019-03-19 13:03:07
>> GMT;
>> > 12s ago
>> >  Docs: man:libvirtd(8)
>> >https://libvirt.org
>> >   Process: 892 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited,
>> > status=1/FAILURE)
>> >  Main PID: 892 (code=exited, status=1/FAILURE)
>> > Tasks: 19 (limit: 32768)
>> >CGroup: /system.slice/libvirtd.service
>> >├─11338 /usr/sbin/libvirtd -d -l
>> >├─11909 /usr/sbin/dnsmasq
>> > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
>> > --dhcp-script=/usr/libexec/libvirt_leaseshelper
>> >└─11910 /usr/sbin/dnsmasq
>> > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
>> > --dhcp-script=/usr/libexec/libvirt_leaseshelper
>> >
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start
>> > Virtualization daemon.
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit
>> > libvirtd.service entered failed state.
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
>> libvirtd.service
>> > failed.
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
>> libvirtd.service
>> > holdoff time over, scheduling restart.
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Stopped
>> > Virtualization daemon.
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: start request
>> > repeated too quickly for libvirtd.service
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start
>> > Virtualization daemon.
>> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit
>> > libvirtd.service entered failed state.
>> > Mar 19

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

I've managed to make libvirtd running
Now cloudstack console shows both hosts - running

But now as I have removed network, VMs are unable to start.

How can I recreate the network now?

On Tue, Mar 19, 2019 at 3:14 PM Ivan Kudryavtsev 
wrote:

> Jevgeniy, it may be a documentation bug. Take s look:
> https://github.com/apache/cloudstack-documentation/pull/27/files
>
> вт, 19 мар. 2019 г., 9:09 Jevgeni Zolotarjov :
>
> > That's it - libvirtd failed to start on second host.
> > Tried restarting, but it does not start.
> >
> >
> > >> Do you have some NUMA constraints or anything which requires
> particular
> > RAM configuration?
> > No
> >
> >  libvirtd.service - Virtualization daemon
> >Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
> > vendor preset: enabled)
> >Active: failed (Result: start-limit) since Tue 2019-03-19 13:03:07
> GMT;
> > 12s ago
> >  Docs: man:libvirtd(8)
> >https://libvirt.org
> >   Process: 892 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited,
> > status=1/FAILURE)
> >  Main PID: 892 (code=exited, status=1/FAILURE)
> > Tasks: 19 (limit: 32768)
> >CGroup: /system.slice/libvirtd.service
> >├─11338 /usr/sbin/libvirtd -d -l
> >├─11909 /usr/sbin/dnsmasq
> > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
> > --dhcp-script=/usr/libexec/libvirt_leaseshelper
> >└─11910 /usr/sbin/dnsmasq
> > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
> > --dhcp-script=/usr/libexec/libvirt_leaseshelper
> >
> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start
> > Virtualization daemon.
> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit
> > libvirtd.service entered failed state.
> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: libvirtd.service
> > failed.
> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: libvirtd.service
> > holdoff time over, scheduling restart.
> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Stopped
> > Virtualization daemon.
> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: start request
> > repeated too quickly for libvirtd.service
> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start
> > Virtualization daemon.
> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit
> > libvirtd.service entered failed state.
> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: libvirtd.service
> > failed.
> >
> >
> > On Tue, Mar 19, 2019 at 3:04 PM Paul Angus 
> > wrote:
> >
> > > Can you check that the cloudstack agent is running on the host and the
> > > agent logs (usual logs directory)
> > > Also worth checking that libvirt has started ok.  Do you have some NUMA
> > > constraints or anything which requires particular RAM configuration?
> > >
> > > paul.an...@shapeblue.com
> > > www.shapeblue.com
> > > Amadeus House, Floral Street, London  WC2E 9DPUK
> > > @shapeblue
> > >
> > >
> > >
> > >
> > > -Original Message-
> > > From: Jevgeni Zolotarjov 
> > > Sent: 19 March 2019 14:49
> > > To: users@cloudstack.apache.org
> > > Subject: Re: Disaster after maintenance
> > >
> > > Can you try migrating a VM to the server that you changed the RAM
> amount?
> > >
> > > Also:
> > > What is the hypervisor version?
> > > KVM
> > > QEMU Version : 2.0.0
> > > Release : 1.el7.6
> > >
> > >
> > > Host status in ACS?
> > > 1st server: Unsecure
> > > 2nd server: Disconnected
> > >
> > > Did you try to force a VM to start/deploy in this server where you
> > changed
> > > the RAM?
> > > Host status became disconnected. I don't know how to make it
> "connected"
> > > again
> > >
> > >
> > >
> > > On Tue, Mar 19, 2019 at 2:42 PM Rafael Weingärtner <
> > > rafaelweingart...@gmail.com> wrote:
> > >
> > > > Can you try migrating a VM to the server that you changed the RAM
> > amount?
> > > >
> > > > Also:
> > > > What is the hypervisor version?
> > > > Host status in ACS?
> > > > Did you try to force a VM to start/deploy in this server where you
> > > > changed the RAM?
> > > >
> > > >
> > > > On Tue, Mar 19, 2019 at 9:3

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

That's it - libvirtd failed to start on second host.
Tried restarting, but it does not start.


>> Do you have some NUMA constraints or anything which requires particular
RAM configuration?
No

 libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
vendor preset: enabled)
   Active: failed (Result: start-limit) since Tue 2019-03-19 13:03:07 GMT;
12s ago
 Docs: man:libvirtd(8)
   https://libvirt.org
  Process: 892 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited,
status=1/FAILURE)
 Main PID: 892 (code=exited, status=1/FAILURE)
Tasks: 19 (limit: 32768)
   CGroup: /system.slice/libvirtd.service
   ├─11338 /usr/sbin/libvirtd -d -l
   ├─11909 /usr/sbin/dnsmasq
--conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
--dhcp-script=/usr/libexec/libvirt_leaseshelper
   └─11910 /usr/sbin/dnsmasq
--conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
--dhcp-script=/usr/libexec/libvirt_leaseshelper

Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start
Virtualization daemon.
Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit
libvirtd.service entered failed state.
Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: libvirtd.service
failed.
Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: libvirtd.service
holdoff time over, scheduling restart.
Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Stopped
Virtualization daemon.
Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: start request
repeated too quickly for libvirtd.service
Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start
Virtualization daemon.
Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit
libvirtd.service entered failed state.
Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: libvirtd.service
failed.


On Tue, Mar 19, 2019 at 3:04 PM Paul Angus  wrote:

> Can you check that the cloudstack agent is running on the host and the
> agent logs (usual logs directory)
> Also worth checking that libvirt has started ok.  Do you have some NUMA
> constraints or anything which requires particular RAM configuration?
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London  WC2E 9DPUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: Jevgeni Zolotarjov 
> Sent: 19 March 2019 14:49
> To: users@cloudstack.apache.org
> Subject: Re: Disaster after maintenance
>
> Can you try migrating a VM to the server that you changed the RAM amount?
>
> Also:
> What is the hypervisor version?
> KVM
> QEMU Version : 2.0.0
> Release : 1.el7.6
>
>
> Host status in ACS?
> 1st server: Unsecure
> 2nd server: Disconnected
>
> Did you try to force a VM to start/deploy in this server where you changed
> the RAM?
> Host status became disconnected. I don't know how to make it "connected"
> again
>
>
>
> On Tue, Mar 19, 2019 at 2:42 PM Rafael Weingärtner <
> rafaelweingart...@gmail.com> wrote:
>
> > Can you try migrating a VM to the server that you changed the RAM amount?
> >
> > Also:
> > What is the hypervisor version?
> > Host status in ACS?
> > Did you try to force a VM to start/deploy in this server where you
> > changed the RAM?
> >
> >
> > On Tue, Mar 19, 2019 at 9:39 AM Jevgeni Zolotarjov
> >  > >
> > wrote:
> >
> > > We have Cloudstack 4.11.2 setup running fine for few months (>4) The
> > > setup is very simple: 2 hosts We decided to do a maintenance to
> > > increase RAM on both servers
> > >
> > > For this we put first server to maintenance. All VMS moved to second
> > > host after a while.
> > >
> > > Then first server was shutdown, RAM increased, server turned ON.
> > > Now nothing starts on first server.
> > >
> > >
> > > Tried to delete network, but this fails as well
> > >
> > > Please help !
> > >
> > > Here is extract from log:
> > > ==
> > > 2019-03-19 12:27:53,064 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > > (secstorage-1:ctx-16d6c797) (logid:7e3160ce) Zone 1 is ready to
> > > launch secondary storage VM
> > > 2019-03-19 12:27:53,125 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-cbd034b9) (logid:0a8c8bf4) Zone 1 is ready to
> > > launch console proxy
> > > 2019-03-19 12:27:53,181 DEBUG [c.c.a.ApiServlet]
> > > (qtp510113906-285:ctx-6c5e11c3) (logid:cd8e30be) ===START===
> > 192.168.5.140
> > > -- GET
> > >
> > >
> > command=deleteNetwork=4ba834ed-48f3-468f-b667-9bb2d2c258f1
&

Re: Disaster after maintenance

[Jevgeni Zolotarjov]

Can you try migrating a VM to the server that you changed the RAM amount?

Also:
What is the hypervisor version?
KVM
QEMU Version : 2.0.0
Release : 1.el7.6


Host status in ACS?
1st server: Unsecure
2nd server: Disconnected

Did you try to force a VM to start/deploy in this server where you changed
the RAM?
Host status became disconnected. I don't know how to make it "connected"
again



On Tue, Mar 19, 2019 at 2:42 PM Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

> Can you try migrating a VM to the server that you changed the RAM amount?
>
> Also:
> What is the hypervisor version?
> Host status in ACS?
> Did you try to force a VM to start/deploy in this server where you changed
> the RAM?
>
>
> On Tue, Mar 19, 2019 at 9:39 AM Jevgeni Zolotarjov  >
> wrote:
>
> > We have Cloudstack 4.11.2 setup running fine for few months (>4)
> > The setup is very simple: 2 hosts
> > We decided to do a maintenance to increase RAM on both servers
> >
> > For this we put first server to maintenance. All VMS moved to second host
> > after a while.
> >
> > Then first server was shutdown, RAM increased, server turned ON.
> > Now nothing starts on first server.
> >
> >
> > Tried to delete network, but this fails as well
> >
> > Please help !
> >
> > Here is extract from log:
> > ==
> > 2019-03-19 12:27:53,064 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-16d6c797) (logid:7e3160ce) Zone 1 is ready to launch
> > secondary storage VM
> > 2019-03-19 12:27:53,125 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-cbd034b9) (logid:0a8c8bf4) Zone 1 is ready to launch
> > console proxy
> > 2019-03-19 12:27:53,181 DEBUG [c.c.a.ApiServlet]
> > (qtp510113906-285:ctx-6c5e11c3) (logid:cd8e30be) ===START===
> 192.168.5.140
> > -- GET
> >
> >
> command=deleteNetwork=4ba834ed-48f3-468f-b667-9bb2d2c258f1=json&_=1552998473154
> > 2019-03-19 12:27:53,186 DEBUG [c.c.a.ApiServer]
> > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) CIDRs from
> > which account 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is
> allowed
> > to perform API calls: 0.0.0.0/0,::/0
> > 2019-03-19 12:27:53,208 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
> > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:f6751fa7) Add job-5081
> > into job monitoring
> > 2019-03-19 12:27:53,209 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) submit
> async
> > job-5081, details: AsyncJobVO {id:5081, userId: 2, accountId: 2,
> > instanceType: Network, instanceId: 204, cmd:
> > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
> >
> >
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
> >
> >
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1552998473154"},
> > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > result: null, initMsid: 264216221068220, completeMsid: null, lastUpdated:
> > null, lastPolled: null, created: null}
> > 2019-03-19 12:27:53,211 DEBUG [c.c.a.ApiServlet]
> > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) ===END===
> > 192.168.5.140 -- GET
> >
> >
> command=deleteNetwork=4ba834ed-48f3-468f-b667-9bb2d2c258f1=json&_=1552998473154
> > 2019-03-19 12:27:53,212 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Executing
> > AsyncJobVO {id:5081, userId: 2, accountId: 2, instanceType: Network,
> > instanceId: 204, cmd:
> > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
> >
> >
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
> >
> >
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType&quo

Disaster after maintenance

[Jevgeni Zolotarjov]

We have Cloudstack 4.11.2 setup running fine for few months (>4)
The setup is very simple: 2 hosts
We decided to do a maintenance to increase RAM on both servers

For this we put first server to maintenance. All VMS moved to second host
after a while.

Then first server was shutdown, RAM increased, server turned ON.
Now nothing starts on first server.


Tried to delete network, but this fails as well

Please help !

Here is extract from log:
==
2019-03-19 12:27:53,064 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-16d6c797) (logid:7e3160ce) Zone 1 is ready to launch
secondary storage VM
2019-03-19 12:27:53,125 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-cbd034b9) (logid:0a8c8bf4) Zone 1 is ready to launch
console proxy
2019-03-19 12:27:53,181 DEBUG [c.c.a.ApiServlet]
(qtp510113906-285:ctx-6c5e11c3) (logid:cd8e30be) ===START===  192.168.5.140
-- GET
command=deleteNetwork=4ba834ed-48f3-468f-b667-9bb2d2c258f1=json&_=1552998473154
2019-03-19 12:27:53,186 DEBUG [c.c.a.ApiServer]
(qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) CIDRs from
which account 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is allowed
to perform API calls: 0.0.0.0/0,::/0
2019-03-19 12:27:53,208 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:f6751fa7) Add job-5081
into job monitoring
2019-03-19 12:27:53,209 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) submit async
job-5081, details: AsyncJobVO {id:5081, userId: 2, accountId: 2,
instanceType: Network, instanceId: 204, cmd:
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
{"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1552998473154"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 264216221068220, completeMsid: null, lastUpdated:
null, lastPolled: null, created: null}
2019-03-19 12:27:53,211 DEBUG [c.c.a.ApiServlet]
(qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) ===END===
192.168.5.140 -- GET
command=deleteNetwork=4ba834ed-48f3-468f-b667-9bb2d2c258f1=json&_=1552998473154
2019-03-19 12:27:53,212 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Executing
AsyncJobVO {id:5081, userId: 2, accountId: 2, instanceType: Network,
instanceId: 204, cmd:
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
{"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1552998473154"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 264216221068220, completeMsid: null, lastUpdated:
null, lastPolled: null, created: null}
2019-03-19 12:27:53,257 WARN  [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-1:ctx-d4970c19 job-5081 ctx-d5de7979) (logid:16897ea6)
Can't delete the network, not all user vms are expunged. Vm
VM[User|i-2-11-VM] is in Stopped state
2019-03-19 12:27:53,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Complete async
job-5081, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to delete network"}
2019-03-19 12:27:53,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Publish async
job-5081 complete on message bus
2019-03-19 12:27:53,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Wake up jobs
related to job-5081
2019-03-19 12:27:53,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Update db
status for job-5081
2019-03-19 12:27:53,265 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Wake up jobs
joined with job-5081 and disjoin all subjobs created from job- 5081
2019-03-19 12:27:53,267 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Done executing
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd for job-5081
2019-03-19 12:27:53,267 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) Remove job-5081
from job monitoring
2019-03-19 12:27:56,230 DEBUG [c.c.a.ApiServlet]
(qtp510113906-28:ctx-e6c5bc85) (logid:7fe68f75) 

Re: ***UNCHECKED*** Re: Unable to communicate to instances on new host - iptables?

[Jevgeni Zolotarjov]

Hello

Can you tell me, how do I find if this is my guest network.

This is what I find in configuration for the guestnetwork:
Name defaultGuestNetwork
Type Shared
State Setup
VPC ID N/A
Persistent No
broadcasturi vlan://untagged
Network CIDR
IPv6 Gateway
IPv6 CIDR
Reserved IP Range
Redundant Router No
Network domain cs1cloud.internal


I guess, the answer to your question is NO. But how do I make proper
configuration?

best regards,
Jevgeni


On Wed, Sep 19, 2018 at 4:53 PM Simon Weller 
wrote:

> Is your guest network the bond0.200?
>
>
>
>
> ________
> From: Jevgeni Zolotarjov 
> Sent: Wednesday, September 19, 2018 9:34 AM
> To: users@cloudstack.apache.org
> Subject: Re: Unable to communicate to instances on new host - iptables?
>
> sure
>
> iptables:
> *mangle
> :PREROUTING ACCEPT [4215:32894293]
> :INPUT ACCEPT [3585:32849592]
> :FORWARD ACCEPT [756:57998]
> :OUTPUT ACCEPT [3739:715406]
> :POSTROUTING ACCEPT [4495:773404]
> COMMIT
>
> *nat
> :PREROUTING ACCEPT [22:3593]
> :INPUT ACCEPT [0:0]
> :OUTPUT ACCEPT [3:4508]
> :POSTROUTING ACCEPT [25:8101]
> COMMIT
>
> *filter
> :INPUT ACCEPT [0:0]
> :FORWARD ACCEPT [28:1788]
> :OUTPUT ACCEPT [0:0]
> -A INPUT -p tcp -m tcp --dport 49152:49216 -j ACCEPT
> -A INPUT -p tcp -m tcp --dport 5900:6100 -j ACCEPT
> -A INPUT -p tcp -m tcp --dport 16509 -j ACCEPT
> -A INPUT -p tcp -m tcp --dport 1798 -j ACCEPT
> -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
> -A INPUT -i lo -m comment --comment "Allow all loopback traffic" -j ACCEPT
> -A INPUT -d 127.0.0.0/8 ! -i lo -m comment --comment "Drop all traffic to
> 127 that doesn\'t use lo" -j REJECT --reject-with icmp-port-unreachable
> -A INPUT -m comment --comment "Accept all incoming" -j ACCEPT
> -A INPUT -m state --state RELATED,ESTABLISHED -m comment --comment "Allow
> all incoming on established connections" -j ACCEPT
> -A OUTPUT -m comment --comment "Accept all outgoing" -j ACCEPT
> COMMIT
>
>
> On Wed, Sep 19, 2018 at 5:31 PM Simon Weller 
> wrote:
>
> > Can you provide your iptables rules on your hosts?
> >
> >
> >
> > 
> > From: Jevgeni Zolotarjov 
> > Sent: Wednesday, September 19, 2018 9:29 AM
> > To: users@cloudstack.apache.org
> > Subject: Re: Unable to communicate to instances on new host - iptables?
> >
> > sorry. corrected network config
> >
> > ifcfg-bond0:
> > TYPE=Bond
> > BONDING_MASTER=yes
> > BONDING_OPTS="mode=802.3ad miimon=100 updelay=0 downdelay=0"
> > DEVICE=bond0
> > ONBOOT=yes
> > BOOTPROTO=none
> > USERCTL=no
> > HOTPLUG=no
> > BRIDGE=cloudbr0
> > NM_CONTROLLED=no
> >
> > ifcfg-bond0.200:
> > DEVICE=bond0.200
> > ONBOOT=yes
> > HOTPLUG=no
> > BOOTPROTO=none
> > VLAN=yes
> > BRIDGE=cloudbr1
> >
> >
> > ifcfg-cloudbr0:
> > DEVICE=cloudbr0
> > TYPE=Bridge
> > ONBOOT=yes
> > BOOTPROTO=none
> > IPV6INIT=no
> > IPV6_AUTOCONF=no
> > DELAY=5
> > STP=yes
> > IPADDR=192.168.1.5
> > GATEWAY=192.168.1.1
> > NETMASK=255.255.254.0
> >
> > ifcfg-cloudbr1:
> > DEVICE=cloudbr1
> > TYPE=Bridge
> > ONBOOT=yes
> > BOOTPROTO=none
> > IPV6INIT=no
> > IPV6_AUTOCONF=no
> > DELAY=5
> > STP=yes
> >
> > On Wed, Sep 19, 2018 at 5:27 PM Jevgeni Zolotarjov <
> j.zolotar...@gmail.com
> > >
> > wrote:
> >
> > > Hi Simon,
> > >
> > > I am not using advanced network.
> > >
> > > Here is my network configuration
> > > ifcfg-bond0:
> > > TYPE=Bond
> > > BONDING_MASTER=yes
> > > BONDING_OPTS="mode=802.3ad miimon=100 updelay=0 downdelay=0"
> > > DEVICE=bond0
> > > ONBOOT=yes
> > > BOOTPROTO=none
> > > USERCTL=no
> > > HOTPLUG=no
> > > BRIDGE=cloudbr0
> > > NM_CONTROLLED=no
> > >
> > > ifcfg-bond0.200:
> > > DEVICE=bond0.200
> > > ONBOOT=yes
> > > HOTPLUG=no
> > > BOOTPROTO=none
> > > VLAN=yes
> > > BRIDGE=cloudbr1
> > >
> > > ifcfg-cloudbr0:
> > >
> > > DEVICE=bond0.200
> > > ONBOOT=yes
> > > HOTPLUG=no
> > > BOOTPROTO=none
> > > #TYPE=Ethernet
> > > VLAN=yes
> > > BRIDGE=cloudbr1
> > >
> > > ifcfg-cloudbr0:
> > > DEVICE=cloudbr0
> > > TYPE=Bridge
> > > ONBOOT=yes
> > > BOOTPROTO=none
> > > IPV6INIT=no
> > > IPV6_AUT

Re: Unable to communicate to instances on new host - iptables?

[Jevgeni Zolotarjov]

sure

iptables:
*mangle
:PREROUTING ACCEPT [4215:32894293]
:INPUT ACCEPT [3585:32849592]
:FORWARD ACCEPT [756:57998]
:OUTPUT ACCEPT [3739:715406]
:POSTROUTING ACCEPT [4495:773404]
COMMIT

*nat
:PREROUTING ACCEPT [22:3593]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [3:4508]
:POSTROUTING ACCEPT [25:8101]
COMMIT

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [28:1788]
:OUTPUT ACCEPT [0:0]
-A INPUT -p tcp -m tcp --dport 49152:49216 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 5900:6100 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 16509 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 1798 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i lo -m comment --comment "Allow all loopback traffic" -j ACCEPT
-A INPUT -d 127.0.0.0/8 ! -i lo -m comment --comment "Drop all traffic to
127 that doesn\'t use lo" -j REJECT --reject-with icmp-port-unreachable
-A INPUT -m comment --comment "Accept all incoming" -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -m comment --comment "Allow
all incoming on established connections" -j ACCEPT
-A OUTPUT -m comment --comment "Accept all outgoing" -j ACCEPT
COMMIT


On Wed, Sep 19, 2018 at 5:31 PM Simon Weller 
wrote:

> Can you provide your iptables rules on your hosts?
>
>
>
> 
> From: Jevgeni Zolotarjov 
> Sent: Wednesday, September 19, 2018 9:29 AM
> To: users@cloudstack.apache.org
> Subject: Re: Unable to communicate to instances on new host - iptables?
>
> sorry. corrected network config
>
> ifcfg-bond0:
> TYPE=Bond
> BONDING_MASTER=yes
> BONDING_OPTS="mode=802.3ad miimon=100 updelay=0 downdelay=0"
> DEVICE=bond0
> ONBOOT=yes
> BOOTPROTO=none
> USERCTL=no
> HOTPLUG=no
> BRIDGE=cloudbr0
> NM_CONTROLLED=no
>
> ifcfg-bond0.200:
> DEVICE=bond0.200
> ONBOOT=yes
> HOTPLUG=no
> BOOTPROTO=none
> VLAN=yes
> BRIDGE=cloudbr1
>
>
> ifcfg-cloudbr0:
> DEVICE=cloudbr0
> TYPE=Bridge
> ONBOOT=yes
> BOOTPROTO=none
> IPV6INIT=no
> IPV6_AUTOCONF=no
> DELAY=5
> STP=yes
> IPADDR=192.168.1.5
> GATEWAY=192.168.1.1
> NETMASK=255.255.254.0
>
> ifcfg-cloudbr1:
> DEVICE=cloudbr1
> TYPE=Bridge
> ONBOOT=yes
> BOOTPROTO=none
> IPV6INIT=no
> IPV6_AUTOCONF=no
> DELAY=5
> STP=yes
>
> On Wed, Sep 19, 2018 at 5:27 PM Jevgeni Zolotarjov  >
> wrote:
>
> > Hi Simon,
> >
> > I am not using advanced network.
> >
> > Here is my network configuration
> > ifcfg-bond0:
> > TYPE=Bond
> > BONDING_MASTER=yes
> > BONDING_OPTS="mode=802.3ad miimon=100 updelay=0 downdelay=0"
> > DEVICE=bond0
> > ONBOOT=yes
> > BOOTPROTO=none
> > USERCTL=no
> > HOTPLUG=no
> > BRIDGE=cloudbr0
> > NM_CONTROLLED=no
> >
> > ifcfg-bond0.200:
> > DEVICE=bond0.200
> > ONBOOT=yes
> > HOTPLUG=no
> > BOOTPROTO=none
> > VLAN=yes
> > BRIDGE=cloudbr1
> >
> > ifcfg-cloudbr0:
> >
> > DEVICE=bond0.200
> > ONBOOT=yes
> > HOTPLUG=no
> > BOOTPROTO=none
> > #TYPE=Ethernet
> > VLAN=yes
> > BRIDGE=cloudbr1
> >
> > ifcfg-cloudbr0:
> > DEVICE=cloudbr0
> > TYPE=Bridge
> > ONBOOT=yes
> > BOOTPROTO=none
> > IPV6INIT=no
> > IPV6_AUTOCONF=no
> > DELAY=5
> > STP=yes
> > IPADDR=192.168.1.5
> > GATEWAY=192.168.1.1
> > NETMASK=255.255.254.0
> >
> > ifcfg-cloudbr1:
> > DEVICE=cloudbr1
> > TYPE=Bridge
> > ONBOOT=yes
> > BOOTPROTO=none
> > IPV6INIT=no
> > IPV6_AUTOCONF=no
> > DELAY=5
> > STP=yes
> >
> >
> >
> > On Wed, Sep 19, 2018 at 3:10 PM Simon Weller 
> > wrote:
> >
> >> Jevgeni,
> >>
> >>
> >> What type of networking are you using on your hosts? If advanced, what
> >> type of isolation?
> >>
> >>
> >> - Si
> >>
> >> 
> >> From: Jevgeni Zolotarjov 
> >> Sent: Wednesday, September 19, 2018 3:17 AM
> >> To: users@cloudstack.apache.org
> >> Subject: Unable to communicate to instances on new host - iptables?
> >>
> >> Hello!
> >>
> >> We are running CS 4.11.1 on CentOS7 (latest)
> >>
> >> Previously the installation had just 1 KVM host.
> >> Now we added another identical host.
> >> After some configuration hassle with libvirtd, new host is up and
> running.
> >>
> >> I followed strictly the host installation guide for 4.11.
> >> But instances running on new host are not accessible via tcp/ip. Neither
> >> they can access network.
> >>
> >> I found out that stopping iptables on new host resolves the problem. But
> >> this is not the solution, I guess.
> >>
> >> Please help.
> >>
> >
>

Re: Unable to communicate to instances on new host - iptables?

[Jevgeni Zolotarjov]

sorry. corrected network config

ifcfg-bond0:
TYPE=Bond
BONDING_MASTER=yes
BONDING_OPTS="mode=802.3ad miimon=100 updelay=0 downdelay=0"
DEVICE=bond0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
HOTPLUG=no
BRIDGE=cloudbr0
NM_CONTROLLED=no

ifcfg-bond0.200:
DEVICE=bond0.200
ONBOOT=yes
HOTPLUG=no
BOOTPROTO=none
VLAN=yes
BRIDGE=cloudbr1


ifcfg-cloudbr0:
DEVICE=cloudbr0
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=none
IPV6INIT=no
IPV6_AUTOCONF=no
DELAY=5
STP=yes
IPADDR=192.168.1.5
GATEWAY=192.168.1.1
NETMASK=255.255.254.0

ifcfg-cloudbr1:
DEVICE=cloudbr1
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=none
IPV6INIT=no
IPV6_AUTOCONF=no
DELAY=5
STP=yes

On Wed, Sep 19, 2018 at 5:27 PM Jevgeni Zolotarjov 
wrote:

> Hi Simon,
>
> I am not using advanced network.
>
> Here is my network configuration
> ifcfg-bond0:
> TYPE=Bond
> BONDING_MASTER=yes
> BONDING_OPTS="mode=802.3ad miimon=100 updelay=0 downdelay=0"
> DEVICE=bond0
> ONBOOT=yes
> BOOTPROTO=none
> USERCTL=no
> HOTPLUG=no
> BRIDGE=cloudbr0
> NM_CONTROLLED=no
>
> ifcfg-bond0.200:
> DEVICE=bond0.200
> ONBOOT=yes
> HOTPLUG=no
> BOOTPROTO=none
> VLAN=yes
> BRIDGE=cloudbr1
>
> ifcfg-cloudbr0:
>
> DEVICE=bond0.200
> ONBOOT=yes
> HOTPLUG=no
> BOOTPROTO=none
> #TYPE=Ethernet
> VLAN=yes
> BRIDGE=cloudbr1
>
> ifcfg-cloudbr0:
> DEVICE=cloudbr0
> TYPE=Bridge
> ONBOOT=yes
> BOOTPROTO=none
> IPV6INIT=no
> IPV6_AUTOCONF=no
> DELAY=5
> STP=yes
> IPADDR=192.168.1.5
> GATEWAY=192.168.1.1
> NETMASK=255.255.254.0
>
> ifcfg-cloudbr1:
> DEVICE=cloudbr1
> TYPE=Bridge
> ONBOOT=yes
> BOOTPROTO=none
> IPV6INIT=no
> IPV6_AUTOCONF=no
> DELAY=5
> STP=yes
>
>
>
> On Wed, Sep 19, 2018 at 3:10 PM Simon Weller 
> wrote:
>
>> Jevgeni,
>>
>>
>> What type of networking are you using on your hosts? If advanced, what
>> type of isolation?
>>
>>
>> - Si
>>
>> 
>> From: Jevgeni Zolotarjov 
>> Sent: Wednesday, September 19, 2018 3:17 AM
>> To: users@cloudstack.apache.org
>> Subject: Unable to communicate to instances on new host - iptables?
>>
>> Hello!
>>
>> We are running CS 4.11.1 on CentOS7 (latest)
>>
>> Previously the installation had just 1 KVM host.
>> Now we added another identical host.
>> After some configuration hassle with libvirtd, new host is up and running.
>>
>> I followed strictly the host installation guide for 4.11.
>> But instances running on new host are not accessible via tcp/ip. Neither
>> they can access network.
>>
>> I found out that stopping iptables on new host resolves the problem. But
>> this is not the solution, I guess.
>>
>> Please help.
>>
>

Re: Unable to communicate to instances on new host - iptables?

[Jevgeni Zolotarjov]

Hi Simon,

I am not using advanced network.

Here is my network configuration
ifcfg-bond0:
TYPE=Bond
BONDING_MASTER=yes
BONDING_OPTS="mode=802.3ad miimon=100 updelay=0 downdelay=0"
DEVICE=bond0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
HOTPLUG=no
BRIDGE=cloudbr0
NM_CONTROLLED=no

ifcfg-bond0.200:
DEVICE=bond0.200
ONBOOT=yes
HOTPLUG=no
BOOTPROTO=none
VLAN=yes
BRIDGE=cloudbr1

ifcfg-cloudbr0:

DEVICE=bond0.200
ONBOOT=yes
HOTPLUG=no
BOOTPROTO=none
#TYPE=Ethernet
VLAN=yes
BRIDGE=cloudbr1

ifcfg-cloudbr0:
DEVICE=cloudbr0
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=none
IPV6INIT=no
IPV6_AUTOCONF=no
DELAY=5
STP=yes
IPADDR=192.168.1.5
GATEWAY=192.168.1.1
NETMASK=255.255.254.0

ifcfg-cloudbr1:
DEVICE=cloudbr1
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=none
IPV6INIT=no
IPV6_AUTOCONF=no
DELAY=5
STP=yes



On Wed, Sep 19, 2018 at 3:10 PM Simon Weller 
wrote:

> Jevgeni,
>
>
> What type of networking are you using on your hosts? If advanced, what
> type of isolation?
>
>
> - Si
>
> ________
> From: Jevgeni Zolotarjov 
> Sent: Wednesday, September 19, 2018 3:17 AM
> To: users@cloudstack.apache.org
> Subject: Unable to communicate to instances on new host - iptables?
>
> Hello!
>
> We are running CS 4.11.1 on CentOS7 (latest)
>
> Previously the installation had just 1 KVM host.
> Now we added another identical host.
> After some configuration hassle with libvirtd, new host is up and running.
>
> I followed strictly the host installation guide for 4.11.
> But instances running on new host are not accessible via tcp/ip. Neither
> they can access network.
>
> I found out that stopping iptables on new host resolves the problem. But
> this is not the solution, I guess.
>
> Please help.
>

Unable to communicate to instances on new host - iptables?

[Jevgeni Zolotarjov]

Hello!

We are running CS 4.11.1 on CentOS7 (latest)

Previously the installation had just 1 KVM host.
Now we added another identical host.
After some configuration hassle with libvirtd, new host is up and running.

I followed strictly the host installation guide for 4.11.
But instances running on new host are not accessible via tcp/ip. Neither
they can access network.

I found out that stopping iptables on new host resolves the problem. But
this is not the solution, I guess.

Please help.

Re: Unable to migrate instance to new host

[Jevgeni Zolotarjov]

Resolved the problem by following changes in /etc/libvirt/libvirtd.conf

listen_tls=0
listen_tcp=1

Now I can move running instances between hosts!

BUT - new problem
The instances, running on freshly added host are not accessible over TCP
from LAN. When I open console for these instances, then the network is
available, IP address is correct and IP addresses in LAN are accessible
from these instances.

What is still missing?
Help appreciated.

On Tue, Sep 18, 2018 at 11:46 PM Jevgeni Zolotarjov 
wrote:

> My host is running on Centos7.
>
> tried to set "LIBVIRTD_ARGS=-l" in
> /etc/sysconfig/libvirtd
>
> But nothing changed
>
> On Tue, Sep 18, 2018 at 8:42 PM Nicolas Bouige  wrote:
>
>> Hello Jevgeni,
>>
>> Whats is your linux distribution ?
>> On ubuntu 16.04,  We ran into the same problem few month ago and we had
>> to modify the libvirt-bin.service as well.
>>
>> 'ExecStart=/usr/sbin/libvirtd $libvirtd_opts' >>
>> 'ExecStart=/usr/sbin/libvirtd -l $libvirtd_opts'
>>
>> Add the "-l" before $libvirtd_opts
>>
>> Best regards,
>> N.B
>>
>> -Message d'origine-
>> De : Jevgeni Zolotarjov [mailto:j.zolotar...@gmail.com]
>> Envoyé : mardi 18 septembre 2018 18:10
>> À : users@cloudstack.apache.org
>> Objet : Unable to migrate instance to new host
>>
>> We were running cloudstack 4.11.1 with 1 host.
>> Now we added another identical host.
>>
>> The procedure completed successfully.
>>
>> But the attempt to migrate instance to this new host fails with error
>> message:
>>
>> Migration was refused connection to destination:
>> qemu+tcp://A.B.C.D/system.
>> Please check libvirt configuration compatibility and firewall rules on
>> the source and destination hosts.
>>
>> iptables configuration on both hosts is the one suggested here
>> http://docs.cloudstack.apache.org/projects/archived-cloudstack-installation/en/4.11/hypervisor/kvm.html#configuring-the-firewall
>>
>> Please help.
>>
>

Re: Unable to migrate instance to new host

[Jevgeni Zolotarjov]

My host is running on Centos7.

tried to set "LIBVIRTD_ARGS=-l" in
/etc/sysconfig/libvirtd

But nothing changed

On Tue, Sep 18, 2018 at 8:42 PM Nicolas Bouige  wrote:

> Hello Jevgeni,
>
> Whats is your linux distribution ?
> On ubuntu 16.04,  We ran into the same problem few month ago and we had to
> modify the libvirt-bin.service as well.
>
> 'ExecStart=/usr/sbin/libvirtd $libvirtd_opts' >>
> 'ExecStart=/usr/sbin/libvirtd -l $libvirtd_opts'
>
> Add the "-l" before $libvirtd_opts
>
> Best regards,
> N.B
>
> -Message d'origine-
> De : Jevgeni Zolotarjov [mailto:j.zolotar...@gmail.com]
> Envoyé : mardi 18 septembre 2018 18:10
> À : users@cloudstack.apache.org
> Objet : Unable to migrate instance to new host
>
> We were running cloudstack 4.11.1 with 1 host.
> Now we added another identical host.
>
> The procedure completed successfully.
>
> But the attempt to migrate instance to this new host fails with error
> message:
>
> Migration was refused connection to destination: qemu+tcp://A.B.C.D/system.
> Please check libvirt configuration compatibility and firewall rules on the
> source and destination hosts.
>
> iptables configuration on both hosts is the one suggested here
> http://docs.cloudstack.apache.org/projects/archived-cloudstack-installation/en/4.11/hypervisor/kvm.html#configuring-the-firewall
>
> Please help.
>

Unable to migrate instance to new host

[Jevgeni Zolotarjov]

We were running cloudstack 4.11.1 with 1 host.
Now we added another identical host.

The procedure completed successfully.

But the attempt to migrate instance to this new host fails with error
message:

Migration was refused connection to destination: qemu+tcp://A.B.C.D/system.
Please check libvirt configuration compatibility and firewall rules on the
source and destination hosts.

iptables configuration on both hosts is the one suggested here
http://docs.cloudstack.apache.org/projects/archived-cloudstack-installation/en/4.11/hypervisor/kvm.html#configuring-the-firewall

Please help.

Re: qemu update 1.5.3 -> newer?

[Jevgeni Zolotarjov]

Yes.

Followed Simon's suggestion. Worked flawlessly.

Thanks Simon!

On Thu, Sep 13, 2018 at 12:02 PM Darius Kasparavičius 
wrote:

> Hello,
>
>
> Simon explained it better than me. We are running qemu-kvm-ev with
> 4.9.x cloudstack for about a year and had no issue with. We moved to
> it due to better ceph performance and lower latencies. Before moving
> all your servers test them for performance an stability. On Wed, Sep
> 12, 2018 at 3:23 PM Jevgeni Zolotarjov  wrote:
> >
> > Doesn't look to be active. The page has zero information about my
> questions
> > and is not maintained either.
> >
> > Other hints?
> >
> > On Wed, Sep 12, 2018 at 1:35 PM Darius Kasparavičius 
> > wrote:
> >
> > > There is a SIG for centos that is running newer versions of
> > > qemu/libvirt. You can find more info about it here
> > > https://wiki.centos.org/SpecialInterestGroup/Virtualization.
> > > On Wed, Sep 12, 2018 at 10:17 AM Jevgeni Zolotarjov
> > >  wrote:
> > > >
> > > > Hi
> > > >
> > > > Our cloudstack 4.11.1 installation works with qemu-kvm 1.5.3, which
> is
> > > > shipped with standard Centos7.
> > > > Anyone has a guide about safe way to upgrade to a newer version of
> qemu?
> > > >
> > > > Best regards,
> > > > Jevgeni
> > >
>

Re: qemu update 1.5.3 -> newer?

[Jevgeni Zolotarjov]

Doesn't look to be active. The page has zero information about my questions
and is not maintained either.

Other hints?

On Wed, Sep 12, 2018 at 1:35 PM Darius Kasparavičius 
wrote:

> There is a SIG for centos that is running newer versions of
> qemu/libvirt. You can find more info about it here
> https://wiki.centos.org/SpecialInterestGroup/Virtualization.
> On Wed, Sep 12, 2018 at 10:17 AM Jevgeni Zolotarjov
>  wrote:
> >
> > Hi
> >
> > Our cloudstack 4.11.1 installation works with qemu-kvm 1.5.3, which is
> > shipped with standard Centos7.
> > Anyone has a guide about safe way to upgrade to a newer version of qemu?
> >
> > Best regards,
> > Jevgeni
>

qemu update 1.5.3 -> newer?

[Jevgeni Zolotarjov]

Hi

Our cloudstack 4.11.1 installation works with qemu-kvm 1.5.3, which is
shipped with standard Centos7.
Anyone has a guide about safe way to upgrade to a newer version of qemu?

Best regards,
Jevgeni

Re: 4.11.0 -> 4.11.1 problem: Guest VMs losing connection after few minutes

[Jevgeni Zolotarjov]

Yes,

But isn't it what is written here
http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.11.1.0/upgrade/upgrade-4.11.html
?

On Fri, Jul 20, 2018 at 3:29 PM Makrand  wrote:

> I think you must have tried to register system VM template from template
> menu from the left side manually. It will be registered as USER only in
> that case.
>
> --
> Makrand
>
>
> On Fri, Jul 20, 2018 at 5:41 PM, Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> wrote:
>
> > Eventually I fixed the problem, without clear understanding of the root
> > cause.
> >
> > I destroyed routerVM, and cloudstack recreated it. But I discovered, that
> > it is version 4.11.0. Not 4.11.1!
> > I checked and systemVM template for 4.11.1 is registered in cloudstack -
> > all OK.
> > I noticed however, that its "Type" property is USER, not SYSTEM.
> >
> > Then I wanted to delete template for 4.11.0, but cloudstack does not
> offer
> > me this option. I can only add new ones.
> >
> > So, I ended with manipulations in the DB itself to make template for
> 4.11.1
> > the only one in the system and have Type = SYSTEM.
> >
> > After that I destroyed again routerVM. It was recreated and it is 4.11.1.
> > And now everything works fine for over an hour already.
> >
> > I hope, thats it.
> >
> > On Fri, Jul 20, 2018 at 12:10 PM ilya musayev <
> > ilya.mailing.li...@gmail.com>
> > wrote:
> >
> > > Have you tried destroying router vm and let CloudStack create new one ?
> > >
> > > On Fri, Jul 20, 2018 at 1:33 AM Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com
> > > >
> > > wrote:
> > >
> > > > - an ip-address conflict.
> > > >   JZ: unlikely, but not impossible. I tried to restart router VM in
> > > > Network-Guest networks -> defaultGuestNetwork -> VirtualAppliances
> > > > While rebooting ping to this router VM disappeared. Hence, no other
> > > device
> > > > is using the same IP.
> > > > But!!! when this virtual router started, then network connection to
> all
> > > > guest VMs disappeared. So, it must be something with this virtual
> > router.
> > > >
> > > > - flakey hardware being one of
> > > > -+ if card in the host
> > > > JZ: higly unlikely
> > > >
> > > > -+ a router with bad firmware
> > > > JZ: also unlikely
> > > >
> > > > - of course a strange cofiguration of the software router in you host
> > > might
> > > > be the issue as well
> > > > JZ: I didnt do any special configuration. Just used default.
> > > >
> > > > by all I know this happening after upgrade sounds like an unhappy
> > > incident
> > > > but can't be sure.
> > > > The iptables restart, was this on the VirtualRouter or on the host,
> or
> > > > maybe on the guest? and the restart network?
> > > >
> > > > JZ: iptables restart on host machine. (or network restart on host)
> > > >
> > > >
> > > >
> > > > On Fri, Jul 20, 2018 at 11:14 AM Daan Hoogland <
> > daan.hoogl...@gmail.com>
> > > > wrote:
> > > >
> > > > > that behaviour sound familiar from a couple of cases:
> > > > > - an ip-address conflict.
> > > > > - flakey hardware being one of
> > > > > -+ if card in the host
> > > > > -+ a router with bad firmware
> > > > > - of course a strange cofiguration of the software router in you
> host
> > > > might
> > > > > be the issue as well
> > > > >
> > > > > by all I know this happening after upgrade sounds like an unhappy
> > > > incident
> > > > > but can't be sure.
> > > > > The iptables restart, was this on the VirtualRouter or on the host,
> > or
> > > > > maybe on the guest? and the restart network?
> > > > >
> > > > >
> > > > > On Fri, Jul 20, 2018 at 7:43 AM, Jevgeni Zolotarjov <
> > > > > j.zolotar...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I updated cloudstack 4.11.0 -> 4.11.1
> > > > > >
> > > > > > Everything went OK during update, but after host reboot guest VMs
> > > lost
> > > > > > connection after few minutes of normal work.
> > > > > > I tried restarting network - systemctl restart network.service
> > > > > > then connection was restored again for few minutes
> > > > > >
> > > > > > Finally I could restore connection by restarting iptables -
> > systemctl
> > > > > > restart iptables.service
> > > > > >
> > > > > > But then again guest VMs lost connection after few minutes of
> > normal
> > > > > > operation.
> > > > > > The time of normal operation can be 5 minutes, but sometimes up
> to
> > 40
> > > > > > minutes.
> > > > > >
> > > > > > Please help me to track the root cause and fix it
> > > > > >
> > > > > > Host OS - Centos 7.5
> > > > > > virtualisation - KVM
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Daan
> > > > >
> > > >
> > >
> >
>

Re: 4.11.0 -> 4.11.1 problem: Guest VMs losing connection after few minutes

[Jevgeni Zolotarjov]

Eventually I fixed the problem, without clear understanding of the root
cause.

I destroyed routerVM, and cloudstack recreated it. But I discovered, that
it is version 4.11.0. Not 4.11.1!
I checked and systemVM template for 4.11.1 is registered in cloudstack -
all OK.
I noticed however, that its "Type" property is USER, not SYSTEM.

Then I wanted to delete template for 4.11.0, but cloudstack does not offer
me this option. I can only add new ones.

So, I ended with manipulations in the DB itself to make template for 4.11.1
the only one in the system and have Type = SYSTEM.

After that I destroyed again routerVM. It was recreated and it is 4.11.1.
And now everything works fine for over an hour already.

I hope, thats it.

On Fri, Jul 20, 2018 at 12:10 PM ilya musayev 
wrote:

> Have you tried destroying router vm and let CloudStack create new one ?
>
> On Fri, Jul 20, 2018 at 1:33 AM Jevgeni Zolotarjov  >
> wrote:
>
> > - an ip-address conflict.
> >   JZ: unlikely, but not impossible. I tried to restart router VM in
> > Network-Guest networks -> defaultGuestNetwork -> VirtualAppliances
> > While rebooting ping to this router VM disappeared. Hence, no other
> device
> > is using the same IP.
> > But!!! when this virtual router started, then network connection to all
> > guest VMs disappeared. So, it must be something with this virtual router.
> >
> > - flakey hardware being one of
> > -+ if card in the host
> > JZ: higly unlikely
> >
> > -+ a router with bad firmware
> > JZ: also unlikely
> >
> > - of course a strange cofiguration of the software router in you host
> might
> > be the issue as well
> > JZ: I didnt do any special configuration. Just used default.
> >
> > by all I know this happening after upgrade sounds like an unhappy
> incident
> > but can't be sure.
> > The iptables restart, was this on the VirtualRouter or on the host, or
> > maybe on the guest? and the restart network?
> >
> > JZ: iptables restart on host machine. (or network restart on host)
> >
> >
> >
> > On Fri, Jul 20, 2018 at 11:14 AM Daan Hoogland 
> > wrote:
> >
> > > that behaviour sound familiar from a couple of cases:
> > > - an ip-address conflict.
> > > - flakey hardware being one of
> > > -+ if card in the host
> > > -+ a router with bad firmware
> > > - of course a strange cofiguration of the software router in you host
> > might
> > > be the issue as well
> > >
> > > by all I know this happening after upgrade sounds like an unhappy
> > incident
> > > but can't be sure.
> > > The iptables restart, was this on the VirtualRouter or on the host, or
> > > maybe on the guest? and the restart network?
> > >
> > >
> > > On Fri, Jul 20, 2018 at 7:43 AM, Jevgeni Zolotarjov <
> > > j.zolotar...@gmail.com>
> > > wrote:
> > >
> > > > I updated cloudstack 4.11.0 -> 4.11.1
> > > >
> > > > Everything went OK during update, but after host reboot guest VMs
> lost
> > > > connection after few minutes of normal work.
> > > > I tried restarting network - systemctl restart network.service
> > > > then connection was restored again for few minutes
> > > >
> > > > Finally I could restore connection by restarting iptables - systemctl
> > > > restart iptables.service
> > > >
> > > > But then again guest VMs lost connection after few minutes of normal
> > > > operation.
> > > > The time of normal operation can be 5 minutes, but sometimes up to 40
> > > > minutes.
> > > >
> > > > Please help me to track the root cause and fix it
> > > >
> > > > Host OS - Centos 7.5
> > > > virtualisation - KVM
> > > >
> > >
> > >
> > >
> > > --
> > > Daan
> > >
> >
>

Re: 4.11.0 -> 4.11.1 problem: Guest VMs losing connection after few minutes

[Jevgeni Zolotarjov]

- an ip-address conflict.
  JZ: unlikely, but not impossible. I tried to restart router VM in
Network-Guest networks -> defaultGuestNetwork -> VirtualAppliances
While rebooting ping to this router VM disappeared. Hence, no other device
is using the same IP.
But!!! when this virtual router started, then network connection to all
guest VMs disappeared. So, it must be something with this virtual router.

- flakey hardware being one of
-+ if card in the host
JZ: higly unlikely

-+ a router with bad firmware
JZ: also unlikely

- of course a strange cofiguration of the software router in you host might
be the issue as well
JZ: I didnt do any special configuration. Just used default.

by all I know this happening after upgrade sounds like an unhappy incident
but can't be sure.
The iptables restart, was this on the VirtualRouter or on the host, or
maybe on the guest? and the restart network?

JZ: iptables restart on host machine. (or network restart on host)



On Fri, Jul 20, 2018 at 11:14 AM Daan Hoogland 
wrote:

> that behaviour sound familiar from a couple of cases:
> - an ip-address conflict.
> - flakey hardware being one of
> -+ if card in the host
> -+ a router with bad firmware
> - of course a strange cofiguration of the software router in you host might
> be the issue as well
>
> by all I know this happening after upgrade sounds like an unhappy incident
> but can't be sure.
> The iptables restart, was this on the VirtualRouter or on the host, or
> maybe on the guest? and the restart network?
>
>
> On Fri, Jul 20, 2018 at 7:43 AM, Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> wrote:
>
> > I updated cloudstack 4.11.0 -> 4.11.1
> >
> > Everything went OK during update, but after host reboot guest VMs lost
> > connection after few minutes of normal work.
> > I tried restarting network - systemctl restart network.service
> > then connection was restored again for few minutes
> >
> > Finally I could restore connection by restarting iptables - systemctl
> > restart iptables.service
> >
> > But then again guest VMs lost connection after few minutes of normal
> > operation.
> > The time of normal operation can be 5 minutes, but sometimes up to 40
> > minutes.
> >
> > Please help me to track the root cause and fix it
> >
> > Host OS - Centos 7.5
> > virtualisation - KVM
> >
>
>
>
> --
> Daan
>

4.11.0 -> 4.11.1 problem: Guest VMs losing connection after few minutes

[Jevgeni Zolotarjov]

I updated cloudstack 4.11.0 -> 4.11.1

Everything went OK during update, but after host reboot guest VMs lost
connection after few minutes of normal work.
I tried restarting network - systemctl restart network.service
then connection was restored again for few minutes

Finally I could restore connection by restarting iptables - systemctl
restart iptables.service

But then again guest VMs lost connection after few minutes of normal
operation.
The time of normal operation can be 5 minutes, but sometimes up to 40
minutes.

Please help me to track the root cause and fix it

Host OS - Centos 7.5
virtualisation - KVM

Re: Is there a way to get back destroyed Virtual Router?

[Jevgeni Zolotarjov]

Thanks Boris!

You saved me.

On Wed, Jul 11, 2018 at 10:28 AM Boris Stoyanov <
boris.stoya...@shapeblue.com> wrote:

> If you restart your network cloudstack will create a new router for you,
> if that does not happen then your zone is not able to deploy a VM and
> you’ll need to dig in the management logs to see whats wrong.
>
> Bobby
>
>
> boris.stoya...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
> > On 11 Jul 2018, at 10:18, Jevgeni Zolotarjov 
> wrote:
> >
> > " But eventually if the network is used Cloudstack will automatically
> > recreate the router"
> >
> > Yes, I hoped for the same. But it doesn't happen. Virtual Router VM has
> not
> > appeared after being destroyed
> >
> > On Wed, Jul 11, 2018 at 10:13 AM Boris Stoyanov <
> > boris.stoya...@shapeblue.com> wrote:
> >
> >> If you’re looking to recover the very same VM, once expunged I don’t
> think
> >> you can. But eventually if the network is used Cloudstack will
> >> automatically recreate the router with the same settings and should be
> >> identical toy the old VM.
> >>
> >> Boris Stoyanov.
> >>
> >>
> >> boris.stoya...@shapeblue.com
> >> www.shapeblue.com
> >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >> @shapeblue
> >>
> >>
> >>
> >>> On 11 Jul 2018, at 10:10, Jevgeni Zolotarjov 
> >> wrote:
> >>>
> >>> I am running cloudstacl 4.11.1
> >>> Virtual Router VM got destroyed unintentionally.
> >>>
> >>> Is there a way to re-create it?
> >>>
> >>> Regards,
> >>> Jevgeni
> >>
> >>
>
>

Re: Is there a way to get back destroyed Virtual Router?

[Jevgeni Zolotarjov]

" But eventually if the network is used Cloudstack will automatically
recreate the router"

Yes, I hoped for the same. But it doesn't happen. Virtual Router VM has not
appeared after being destroyed

On Wed, Jul 11, 2018 at 10:13 AM Boris Stoyanov <
boris.stoya...@shapeblue.com> wrote:

> If you’re looking to recover the very same VM, once expunged I don’t think
> you can. But eventually if the network is used Cloudstack will
> automatically recreate the router with the same settings and should be
> identical toy the old VM.
>
> Boris Stoyanov.
>
>
> boris.stoya...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
> > On 11 Jul 2018, at 10:10, Jevgeni Zolotarjov 
> wrote:
> >
> > I am running cloudstacl 4.11.1
> > Virtual Router VM got destroyed unintentionally.
> >
> > Is there a way to re-create it?
> >
> > Regards,
> > Jevgeni
>
>

Is there a way to get back destroyed Virtual Router?

[Jevgeni Zolotarjov]

I am running cloudstacl 4.11.1
Virtual Router VM got destroyed unintentionally.

Is there a way to re-create it?

Regards,
Jevgeni

Re: cloudstack-management fails to start after upgrade 4.10 -> 4.11

[Jevgeni Zolotarjov]

To finalize, IPs recovered too. I think I started my VMs too early before
router was up.

Now its all fine.

On Fri, Feb 9, 2018 at 6:36 PM, Paul Angus <paul.an...@shapeblue.com> wrote:

> No problem, I'm pleased that you're sorted.
>
>
> Kind regards,
>
> Paul Angus
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: Jevgeni Zolotarjov [mailto:j.zolotar...@gmail.com]
> Sent: 09 February 2018 16:34
> To: users@cloudstack.apache.org
> Subject: Re: cloudstack-management fails to start after upgrade 4.10 ->
> 4.11
>
> destroyed virtual router
>
> it got recreated.
> On NOW. My VMs start!!!
>
> All IPs got changed, but I can manage with that.
>
> Thank you for your support
>
> On Fri, Feb 9, 2018 at 6:21 PM, Paul Angus <paul.an...@shapeblue.com>
> wrote:
>
> > Have you done the same for the virtual routers?
> >
> > I'll look at your log in the meantime
> >
> >
> > Kind regards,
> >
> > Paul Angus
> >
> > paul.an...@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
> >
> >
> >
> >
> > -Original Message-
> > From: Jevgeni Zolotarjov [mailto:j.zolotar...@gmail.com]
> > Sent: 09 February 2018 16:19
> > To: users@cloudstack.apache.org
> > Subject: Re: cloudstack-management fails to start after upgrade 4.10 ->
> > 4.11
> >
> > I destroyed system VMs and they got recreated automatically.
> > They are running. I can verify that by
> > virsh list --all
> >
> > and
> > I can see their console and it sugests that it is Cloudstack 4.11
> systemVM
> >
> > BUT
> > it didn't solve the problem. None of my own VM is listed by "virsh list
> > -all". They do not start.
> > I tried to create new VM, it does not start either, due to to the same
> > problem - insufficient capacity
> >
> > On Fri, Feb 9, 2018 at 6:02 PM, Daan Hoogland <daan.hoogl...@gmail.com>
> > wrote:
> >
> > > listen to Paul, not to me.
> > > He's an operator i'm just impatient
> > >
> > > On Fri, Feb 9, 2018 at 5:00 PM, Paul Angus <paul.an...@shapeblue.com>
> > > wrote:
> > >
> > > > After upgrading the code of the mgmt. server you need upgrade your
> > > > system VMs from the old template to ones using the new templates.
> > > >
> > > > This needs to be done for the SSVM, CPVM and all of your virtual
> > routers.
> > > >
> > > > For the SSVM & CPVM you do this by destroying them and CloudStack
> > > > will recreate them with the new template.
> > > > For the virtual routers, you can go to each of them in turn in the
> > > > UI and there will be a upgrade router button.
> > > >
> > > > You get to these through the infrastructure tab in the UI.
> > > >
> > > >
> > > >
> > > >
> > > > paul.an...@shapeblue.com
> > > > www.shapeblue.com
> > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
> > > >
> > > >
> > > >
> > > >
> > > > -Original Message-
> > > > From: Jevgeni Zolotarjov [mailto:j.zolotar...@gmail.com]
> > > > Sent: 09 February 2018 15:55
> > > > To: users@cloudstack.apache.org
> > > > Subject: Re: cloudstack-management fails to start after upgrade 4.10
> ->
> > > > 4.11
> > > >
> > > > Destroyed VMs? No. Definitely not.
> > > >
> > > > At least I can see them all under web console Home->Instances. Can
> make
> > > > snapshots even
> > > >
> > > > My todays management server log https://www.sendspace.com/
> file/nxxgg0
> > > >
> > > >
> > > >
> > > > On Fri, Feb 9, 2018 at 5:33 PM, Paul Angus <paul.an...@shapeblue.com
> >
> > > > wrote:
> > > >
> > > > > Hi Jevgeni,
> > > > >
> > > > > Can I take you off at a slight tangent...
> > > > >
> > > > > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > > > > (logid:0afb959d) DataCenter id = '1' provided is in avoid set,
> > > > > DeploymentPlanner cannot allocate the VM, returning.
> > > > >
> > > > > Have you destroyed yo

Re: cloudstack-management fails to start after upgrade 4.10 -> 4.11

[Jevgeni Zolotarjov]

destroyed virtual router

it got recreated.
On NOW. My VMs start!!!

All IPs got changed, but I can manage with that.

Thank you for your support

On Fri, Feb 9, 2018 at 6:21 PM, Paul Angus <paul.an...@shapeblue.com> wrote:

> Have you done the same for the virtual routers?
>
> I'll look at your log in the meantime
>
>
> Kind regards,
>
> Paul Angus
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: Jevgeni Zolotarjov [mailto:j.zolotar...@gmail.com]
> Sent: 09 February 2018 16:19
> To: users@cloudstack.apache.org
> Subject: Re: cloudstack-management fails to start after upgrade 4.10 ->
> 4.11
>
> I destroyed system VMs and they got recreated automatically.
> They are running. I can verify that by
> virsh list --all
>
> and
> I can see their console and it sugests that it is Cloudstack 4.11 systemVM
>
> BUT
> it didn't solve the problem. None of my own VM is listed by "virsh list
> -all". They do not start.
> I tried to create new VM, it does not start either, due to to the same
> problem - insufficient capacity
>
> On Fri, Feb 9, 2018 at 6:02 PM, Daan Hoogland <daan.hoogl...@gmail.com>
> wrote:
>
> > listen to Paul, not to me.
> > He's an operator i'm just impatient
> >
> > On Fri, Feb 9, 2018 at 5:00 PM, Paul Angus <paul.an...@shapeblue.com>
> > wrote:
> >
> > > After upgrading the code of the mgmt. server you need upgrade your
> > > system VMs from the old template to ones using the new templates.
> > >
> > > This needs to be done for the SSVM, CPVM and all of your virtual
> routers.
> > >
> > > For the SSVM & CPVM you do this by destroying them and CloudStack
> > > will recreate them with the new template.
> > > For the virtual routers, you can go to each of them in turn in the
> > > UI and there will be a upgrade router button.
> > >
> > > You get to these through the infrastructure tab in the UI.
> > >
> > >
> > >
> > >
> > > paul.an...@shapeblue.com
> > > www.shapeblue.com
> > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
> > >
> > >
> > >
> > >
> > > -Original Message-
> > > From: Jevgeni Zolotarjov [mailto:j.zolotar...@gmail.com]
> > > Sent: 09 February 2018 15:55
> > > To: users@cloudstack.apache.org
> > > Subject: Re: cloudstack-management fails to start after upgrade 4.10 ->
> > > 4.11
> > >
> > > Destroyed VMs? No. Definitely not.
> > >
> > > At least I can see them all under web console Home->Instances. Can make
> > > snapshots even
> > >
> > > My todays management server log https://www.sendspace.com/file/nxxgg0
> > >
> > >
> > >
> > > On Fri, Feb 9, 2018 at 5:33 PM, Paul Angus <paul.an...@shapeblue.com>
> > > wrote:
> > >
> > > > Hi Jevgeni,
> > > >
> > > > Can I take you off at a slight tangent...
> > > >
> > > > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > > > (logid:0afb959d) DataCenter id = '1' provided is in avoid set,
> > > > DeploymentPlanner cannot allocate the VM, returning.
> > > >
> > > > Have you destroyed your system vms so that new 4.11 system vms have
> > > > been deployed?
> > > >
> > > > It may help us if you can paste all of your management log from today
> > > > into https://pastebin.com (or similar) to share with us.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > paul.an...@shapeblue.com
> > > > www.shapeblue.com
> > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
> > > >
> > > >
> > > >
> > > >
> > > > -Original Message-
> > > > From: Jevgeni Zolotarjov [mailto:j.zolotar...@gmail.com]
> > > > Sent: 09 February 2018 15:31
> > > > To: users@cloudstack.apache.org
> > > > Subject: Re: cloudstack-management fails to start after upgrade 4.10
> ->
> > > > 4.11
> > > >
> > > > the same result
> > > >
> > > > [root@mtl1-apphst03 management]# virsh list --all
> > > >  IdName

Re: cloudstack-management fails to start after upgrade 4.10 -> 4.11

[Jevgeni Zolotarjov]

I destroyed system VMs and they got recreated automatically.
They are running. I can verify that by
virsh list --all

and
I can see their console and it sugests that it is Cloudstack 4.11 systemVM

BUT
it didn't solve the problem. None of my own VM is listed by "virsh list
-all". They do not start.
I tried to create new VM, it does not start either, due to to the same
problem - insufficient capacity

On Fri, Feb 9, 2018 at 6:02 PM, Daan Hoogland <daan.hoogl...@gmail.com>
wrote:

> listen to Paul, not to me.
> He's an operator i'm just impatient
>
> On Fri, Feb 9, 2018 at 5:00 PM, Paul Angus <paul.an...@shapeblue.com>
> wrote:
>
> > After upgrading the code of the mgmt. server you need upgrade your system
> > VMs from the old template to ones using the new templates.
> >
> > This needs to be done for the SSVM, CPVM and all of your virtual routers.
> >
> > For the SSVM & CPVM you do this by destroying them and CloudStack will
> > recreate them with the new template.
> > For the virtual routers, you can go to each of them in turn in the UI and
> > there will be a upgrade router button.
> >
> > You get to these through the infrastructure tab in the UI.
> >
> >
> >
> >
> > paul.an...@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> > -Original Message-
> > From: Jevgeni Zolotarjov [mailto:j.zolotar...@gmail.com]
> > Sent: 09 February 2018 15:55
> > To: users@cloudstack.apache.org
> > Subject: Re: cloudstack-management fails to start after upgrade 4.10 ->
> > 4.11
> >
> > Destroyed VMs? No. Definitely not.
> >
> > At least I can see them all under web console Home->Instances. Can make
> > snapshots even
> >
> > My todays management server log https://www.sendspace.com/file/nxxgg0
> >
> >
> >
> > On Fri, Feb 9, 2018 at 5:33 PM, Paul Angus <paul.an...@shapeblue.com>
> > wrote:
> >
> > > Hi Jevgeni,
> > >
> > > Can I take you off at a slight tangent...
> > >
> > > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > > (logid:0afb959d) DataCenter id = '1' provided is in avoid set,
> > > DeploymentPlanner cannot allocate the VM, returning.
> > >
> > > Have you destroyed your system vms so that new 4.11 system vms have
> > > been deployed?
> > >
> > > It may help us if you can paste all of your management log from today
> > > into https://pastebin.com (or similar) to share with us.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > paul.an...@shapeblue.com
> > > www.shapeblue.com
> > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
> > >
> > >
> > >
> > >
> > > -Original Message-
> > > From: Jevgeni Zolotarjov [mailto:j.zolotar...@gmail.com]
> > > Sent: 09 February 2018 15:31
> > > To: users@cloudstack.apache.org
> > > Subject: Re: cloudstack-management fails to start after upgrade 4.10 ->
> > > 4.11
> > >
> > > the same result
> > >
> > > [root@mtl1-apphst03 management]# virsh list --all
> > >  IdName   State
> > > 
> > >  1 v-1-VM running
> > >  2 s-2-VM running
> > >
> > >
> > >
> > > On Fri, Feb 9, 2018 at 5:28 PM, Daan Hoogland <daan.hoogl...@gmail.com
> >
> > > wrote:
> > >
> > > >  so try
> > > > # virsh list --all
> > > >
> > > > On Fri, Feb 9, 2018 at 4:27 PM, Jevgeni Zolotarjov
> > > > <j.zolotar...@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > I guess, these are system VMs.
> > > > > none of my own created instances can start.
> > > > >  IdName   State
> > > > > 
> > > > >  1 v-1-VM running
> > > > >  2 s-2-VM running
> > > > >
> > > > > virsh start  3> gives error
> > > > > error: failed to get domain '8'
> > > > > error: Domain not found: no domain with matching name '8'
> 

Re: cloudstack-management fails to start after upgrade 4.10 -> 4.11

[Jevgeni Zolotarjov]

the same result

[root@mtl1-apphst03 management]# virsh list --all
 IdName   State

 1 v-1-VM running
 2 s-2-VM running



On Fri, Feb 9, 2018 at 5:28 PM, Daan Hoogland <daan.hoogl...@gmail.com>
wrote:

>  so try
> # virsh list --all
>
> On Fri, Feb 9, 2018 at 4:27 PM, Jevgeni Zolotarjov <j.zolotar...@gmail.com
> >
> wrote:
>
> > I guess, these are system VMs.
> > none of my own created instances can start.
> >  IdName   State
> > 
> >  1 v-1-VM running
> >  2 s-2-VM running
> >
> > virsh start  3> gives error
> > error: failed to get domain '8'
> > error: Domain not found: no domain with matching name '8'
> >
> >
> > And yes, cloudstack and mysql - everything locally
> >
> >
> >
> > On Fri, Feb 9, 2018 at 5:15 PM, Daan Hoogland <daan.hoogl...@gmail.com>
> > wrote:
> >
> > > On Fri, Feb 9, 2018 at 3:58 PM, Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com
> > > >
> > > wrote:
> > >
> > > > * how to start instance with virsh?
> > > >
> > > ​ah, usually something like
> > > # virsh start 
> > > or
> > > # virsh start <​number>
> > >
> > >
> > > > * starting in debugger? I would rather not do it :)
> > > >
> > > ​ok, i expected as much. It will slow our solution process
> unfortunately
> > > ​
> > >
> > > ​In the below at the bottom it says you have two VMs and both are
> > running.
> > > Is that correct?​
> > >
> > >
> > > > * this is the log around GetHostStatsAnswer
> > > > 2018-02-09 14:05:30,274 DEBUG [c.c.s.StatsCollector]
> > > > (StatsCollector-4:ctx-167407f0) (logid:0c470e95) HostStatsCollector
> is
> > > > running...
> > > > 2018-02-09 14:05:30,322 DEBUG [c.c.a.t.Request]
> > > > (StatsCollector-4:ctx-167407f0) (logid:0c470e95) Seq
> > > 3-613896924205940798:
> > > > Received:  { Ans: , MgmtId: 264216221068220, via: 3(mtl1-apphst03),
> > Ver:
> > > > v1, Flags: 10, { GetHostStatsAnswer } }
> > > > 2018-02-09 14:05:31,388 DEBUG [c.c.s.StatsCollector]
> > > > (StatsCollector-1:ctx-308796ad) (logid:9e87351b) StorageCollector is
> > > > running...
> > > > 2018-02-09 14:05:31,396 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > > > (StatsCollector-1:ctx-308796ad) (logid:9e87351b)
> > > getCommandHostDelegation:
> > > > class com.cloud.agent.api.GetStorageStatsCommand
> > > > 2018-02-09 14:05:31,396 DEBUG [c.c.h.XenServerGuru]
> > > > (StatsCollector-1:ctx-308796ad) (logid:9e87351b) We are returning
> the
> > > > default host to execute commands because the command is not of Copy
> > type.
> > > > 2018-02-09 14:05:31,445 DEBUG [c.c.a.t.Request]
> > > > (StatsCollector-1:ctx-308796ad) (logid:9e87351b) Seq
> > > > 4-1941614389350105109:
> > > > Received:  { Ans: , MgmtId: 264216221068220, via: 4(s-2-VM), Ver: v1,
> > > > Flags: 10, { GetStorageStatsAnswer } }
> > > > 2018-02-09 14:05:31,447 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > > > (StatsCollector-1:ctx-308796ad) (logid:9e87351b)
> > > getCommandHostDelegation:
> > > > class com.cloud.agent.api.GetStorageStatsCommand
> > > > 2018-02-09 14:05:31,447 DEBUG [c.c.h.XenServerGuru]
> > > > (StatsCollector-1:ctx-308796ad) (logid:9e87351b) We are returning
> the
> > > > default host to execute commands because the command is not of Copy
> > type.
> > > > 2018-02-09 14:05:31,517 DEBUG [c.c.a.t.Request]
> > > > (StatsCollector-1:ctx-308796ad) (logid:9e87351b) Seq
> > > 3-613896924205940799:
> > > > Received:  { Ans: , MgmtId: 264216221068220, via: 3(mtl1-apphst03),
> > Ver:
> > > > v1, Flags: 10, { GetStorageStatsAnswer } }
> > > > 2018-02-09 14:05:34,504 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (AsyncJobMgr-Heartbeat-1:ctx-6955e300) (logid:300ab196) Begin
> cleanup
> > > > expired async-jobs
> > > > 2018-02-09 14:05:34,506 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (AsyncJobMgr-Heartbeat-1:ctx-6955e300) (logid:300ab196) End cleanup
> > > > expired
> > > > async-jobs
> > > &g

Re: cloudstack-management fails to start after upgrade 4.10 -> 4.11

[Jevgeni Zolotarjov]

I guess, these are system VMs.
none of my own created instances can start.
 IdName   State

 1 v-1-VM running
 2 s-2-VM running

virsh start  3> gives error
error: failed to get domain '8'
error: Domain not found: no domain with matching name '8'


And yes, cloudstack and mysql - everything locally



On Fri, Feb 9, 2018 at 5:15 PM, Daan Hoogland <daan.hoogl...@gmail.com>
wrote:

> On Fri, Feb 9, 2018 at 3:58 PM, Jevgeni Zolotarjov <j.zolotar...@gmail.com
> >
> wrote:
>
> > * how to start instance with virsh?
> >
> ​ah, usually something like
> # virsh start 
> or
> # virsh start <​number>
>
>
> > * starting in debugger? I would rather not do it :)
> >
> ​ok, i expected as much. It will slow our solution process unfortunately
> ​
>
> ​In the below at the bottom it says you have two VMs and both are running.
> Is that correct?​
>
>
> > * this is the log around GetHostStatsAnswer
> > 2018-02-09 14:05:30,274 DEBUG [c.c.s.StatsCollector]
> > (StatsCollector-4:ctx-167407f0) (logid:0c470e95) HostStatsCollector is
> > running...
> > 2018-02-09 14:05:30,322 DEBUG [c.c.a.t.Request]
> > (StatsCollector-4:ctx-167407f0) (logid:0c470e95) Seq
> 3-613896924205940798:
> > Received:  { Ans: , MgmtId: 264216221068220, via: 3(mtl1-apphst03), Ver:
> > v1, Flags: 10, { GetHostStatsAnswer } }
> > 2018-02-09 14:05:31,388 DEBUG [c.c.s.StatsCollector]
> > (StatsCollector-1:ctx-308796ad) (logid:9e87351b) StorageCollector is
> > running...
> > 2018-02-09 14:05:31,396 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > (StatsCollector-1:ctx-308796ad) (logid:9e87351b)
> getCommandHostDelegation:
> > class com.cloud.agent.api.GetStorageStatsCommand
> > 2018-02-09 14:05:31,396 DEBUG [c.c.h.XenServerGuru]
> > (StatsCollector-1:ctx-308796ad) (logid:9e87351b) We are returning the
> > default host to execute commands because the command is not of Copy type.
> > 2018-02-09 14:05:31,445 DEBUG [c.c.a.t.Request]
> > (StatsCollector-1:ctx-308796ad) (logid:9e87351b) Seq
> > 4-1941614389350105109:
> > Received:  { Ans: , MgmtId: 264216221068220, via: 4(s-2-VM), Ver: v1,
> > Flags: 10, { GetStorageStatsAnswer } }
> > 2018-02-09 14:05:31,447 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > (StatsCollector-1:ctx-308796ad) (logid:9e87351b)
> getCommandHostDelegation:
> > class com.cloud.agent.api.GetStorageStatsCommand
> > 2018-02-09 14:05:31,447 DEBUG [c.c.h.XenServerGuru]
> > (StatsCollector-1:ctx-308796ad) (logid:9e87351b) We are returning the
> > default host to execute commands because the command is not of Copy type.
> > 2018-02-09 14:05:31,517 DEBUG [c.c.a.t.Request]
> > (StatsCollector-1:ctx-308796ad) (logid:9e87351b) Seq
> 3-613896924205940799:
> > Received:  { Ans: , MgmtId: 264216221068220, via: 3(mtl1-apphst03), Ver:
> > v1, Flags: 10, { GetStorageStatsAnswer } }
> > 2018-02-09 14:05:34,504 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-6955e300) (logid:300ab196) Begin cleanup
> > expired async-jobs
> > 2018-02-09 14:05:34,506 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-6955e300) (logid:300ab196) End cleanup
> > expired
> > async-jobs
> > 2018-02-09 14:05:35,979 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-e83be1a3) (logid:60b9de64) Zone 1 is ready to launch
> > secondary storage VM
> > 2018-02-09 14:05:36,002 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-508f06d4) (logid:fea1da44) Zone 1 is ready to launch
> > console proxy
> > 2018-02-09 14:05:36,697 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentManager-Handler-14:null) (logid:) SeqA 5-273: Processing Seq 5-273:
> > { Cmd , MgmtId: -1, via: 5, Ver: v1, Flags: 11,
> > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand"
> > :{"_proxyVmId":1,"_loadInfo":"{\n
> > \"connections\": []\n}","wait":0}}] }
> > 2018-02-09 14:05:36,699 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentManager-Handler-14:null) (logid:) SeqA 5-273: Sending Seq 5-273:  {
> > Ans: , MgmtId: 264216221068220, via: 5, Ver: v1, Flags: 100010,
> > [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> > 2018-02-09 14:05:44,502 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-61f44674) (logid:94748b08) Begin cleanup
> > expired async-jobs
> > 2018-02-09 14:05:44,505 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbea

Re: cloudstack-management fails to start after upgrade 4.10 -> 4.11

[Jevgeni Zolotarjov]

PowerOn
2018-02-09 14:06:16,897 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-4:null) (logid:) VM power state does not change, skip
DB writing. vm id: 1
2018-02-09 14:06:16,897 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-4:null) (logid:) VM state report. host: 3, vm id: 2,
power state: PowerOn
2018-02-09 14:06:16,898 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-4:null) (logid:) VM power state does not change, skip
DB writing. vm id: 2
2018-02-09 14:06:16,899 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-4:null) (logid:) Done with process of VM state
report. host: 3



On Fri, Feb 9, 2018 at 4:47 PM, Daan Hoogland <daan.hoogl...@gmail.com>
wrote:

> and with cloudstack (and mysql) running locally, right?
> So my next step would be to stop cloudstack and see if you can start the
> image with virsh?
> Cloudstack thinks for some reason the host is not a suitable target. I have
> no clue why that is from your log. You can start cloudstack in a debugger
> but given you are asking on users@, i don't think you are familiar with
> that kind of work are, you?
> You might also want to look in the management (and agent) log to see if you
> can find anything about the capacity reported. Look for GetHostStatsAnswer.
>
> On Fri, Feb 9, 2018 at 3:11 PM, Jevgeni Zolotarjov <j.zolotar...@gmail.com
> >
> wrote:
>
> > Yes, its KVM
> >
> > On Fri, Feb 9, 2018 at 4:06 PM, Daan Hoogland <daan.hoogl...@gmail.com>
> > wrote:
> >
> > > I am at a loss but really would like to help you.
> > > is it a KVM with the management server running locally?
> > >
> > > On Fri, Feb 9, 2018 at 3:02 PM, Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com
> > > >
> > > wrote:
> > >
> > > > My Host is only 1 machine atm:
> > > > Dell PowerEdge610
> > > > OS: CentOS7 (latest)
> > > > CPU: 2x12 cores (24 cores in total)
> > > > RAM: 192 GB
> > > > Storage: 3+ TB
> >
>
>
> --
> Daan
>

Re: cloudstack-management fails to start after upgrade 4.10 -> 4.11

[Jevgeni Zolotarjov]

Yes, its KVM

On Fri, Feb 9, 2018 at 4:06 PM, Daan Hoogland <daan.hoogl...@gmail.com>
wrote:

> I am at a loss but really would like to help you.
> is it a KVM with the management server running locally?
>
> On Fri, Feb 9, 2018 at 3:02 PM, Jevgeni Zolotarjov <j.zolotar...@gmail.com
> >
> wrote:
>
> > My Host is only 1 machine atm:
> > Dell PowerEdge610
> > OS: CentOS7 (latest)
> > CPU: 2x12 cores (24 cores in total)
> > RAM: 192 GB
> > Storage: 3+ TB
> >
> > I was running cloudstack-4.10 without problems since it was released
> >
> > Your help is appreciated
> >
> > On Fri, Feb 9, 2018 at 3:57 PM, Daan Hoogland <daan.hoogl...@gmail.com>
> > wrote:
> >
> > > Jevgeni,
> > >
> > > can you describe your environment in more detail? the host[3] that is
> in
> > > the avoid list, is it the host that VM was/is originally running on? Is
> > > anything running on it atm? what is it's distribution and version?
> > etc
> > >
> > > On Fri, Feb 9, 2018 at 2:44 PM, Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com
> > > >
> > > wrote:
> > >
> > > > To follow up:
> > > > this is what I have in log:
> > > > 2018-02-09 13:31:48,836 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > > > (logid:0afb959d) DeploymentPlanner allocation algorithm: null
> > > > 2018-02-09 13:31:48,836 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > > > (logid:0afb959d) Trying to allocate a host and storage pools from
> dc:1,
> > > > pod:1,cluster:null, requested cpu: 500, requested ram: 2147483648
> > > > 2018-02-09 13:31:48,836 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > > > (logid:0afb959d) Is ROOT volume READY (pool already allocated)?: No
> > > > 2018-02-09 13:31:48,840 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > > > (logid:0afb959d) Deploy avoids pods: [], clusters: [], hosts: [3]
> > > > 2018-02-09 13:31:48,840 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > > > (logid:0afb959d) DataCenter id = '1' provided is in avoid set,
> > > > DeploymentPlanner cannot allocate the VM, returning.
> > > > 2018-02-09 13:31:48,843 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > > > (logid:0afb959d) VM state transitted from :Starting to Stopped with
> > > event:
> > > > OperationFailedvm's original host id: 3 new host id: null host id
> > before
> > > > state transition: 3
> > > > 2018-02-09 13:31:48,847 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > > > (logid:0afb959d) Hosts's actual total CPU: 70224 and CPU after
> applying
> > > > overprovisioning: 70224
> > > > 2018-02-09 13:31:48,847 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > > > (logid:0afb959d) Hosts's actual total RAM: 205071147008 and RAM after
> > > > applying overprovisioning: 205071155200
> > > > 2018-02-09 13:31:48,847 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > > > (logid:0afb959d) release cpu from host: 3, old used: 1500,reserved:
> > > 26100,
> > > > actual total: 70224, total with overprovisioning: 70224; new used:
> > > > 1000,reserved:26100; movedfromreserved: false,moveToReserveredfalse
> > > > 2018-02-09 13:31:48,847 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > > > (logid:0afb959d) release mem from host: 3, old used:
> > 3758096384,reserved:
> > > > 81335943168, total: 205071155200; new used:
> > > > 1610612736,reserved:81335943168; movedfromreserved:
> > > > false,moveToReserveredfalse
> > > > 2018-02-09 13:31:48,854 ERROR [c.c.v.VmWorkJobHandlerProxy]
> > > > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > > > (logid:0afb959d) Invocation exception, caused by:
> > > > co

Re: cloudstack-management fails to start after upgrade 4.10 -> 4.11

[Jevgeni Zolotarjov]

My Host is only 1 machine atm:
Dell PowerEdge610
OS: CentOS7 (latest)
CPU: 2x12 cores (24 cores in total)
RAM: 192 GB
Storage: 3+ TB

I was running cloudstack-4.10 without problems since it was released

Your help is appreciated

On Fri, Feb 9, 2018 at 3:57 PM, Daan Hoogland <daan.hoogl...@gmail.com>
wrote:

> Jevgeni,
>
> can you describe your environment in more detail? the host[3] that is in
> the avoid list, is it the host that VM was/is originally running on? Is
> anything running on it atm? what is it's distribution and version? etc
>
> On Fri, Feb 9, 2018 at 2:44 PM, Jevgeni Zolotarjov <j.zolotar...@gmail.com
> >
> wrote:
>
> > To follow up:
> > this is what I have in log:
> > 2018-02-09 13:31:48,836 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > (logid:0afb959d) DeploymentPlanner allocation algorithm: null
> > 2018-02-09 13:31:48,836 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > (logid:0afb959d) Trying to allocate a host and storage pools from dc:1,
> > pod:1,cluster:null, requested cpu: 500, requested ram: 2147483648
> > 2018-02-09 13:31:48,836 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > (logid:0afb959d) Is ROOT volume READY (pool already allocated)?: No
> > 2018-02-09 13:31:48,840 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > (logid:0afb959d) Deploy avoids pods: [], clusters: [], hosts: [3]
> > 2018-02-09 13:31:48,840 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > (logid:0afb959d) DataCenter id = '1' provided is in avoid set,
> > DeploymentPlanner cannot allocate the VM, returning.
> > 2018-02-09 13:31:48,843 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > (logid:0afb959d) VM state transitted from :Starting to Stopped with
> event:
> > OperationFailedvm's original host id: 3 new host id: null host id before
> > state transition: 3
> > 2018-02-09 13:31:48,847 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > (logid:0afb959d) Hosts's actual total CPU: 70224 and CPU after applying
> > overprovisioning: 70224
> > 2018-02-09 13:31:48,847 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > (logid:0afb959d) Hosts's actual total RAM: 205071147008 and RAM after
> > applying overprovisioning: 205071155200
> > 2018-02-09 13:31:48,847 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > (logid:0afb959d) release cpu from host: 3, old used: 1500,reserved:
> 26100,
> > actual total: 70224, total with overprovisioning: 70224; new used:
> > 1000,reserved:26100; movedfromreserved: false,moveToReserveredfalse
> > 2018-02-09 13:31:48,847 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > (logid:0afb959d) release mem from host: 3, old used: 3758096384,reserved:
> > 81335943168, total: 205071155200; new used:
> > 1610612736,reserved:81335943168; movedfromreserved:
> > false,moveToReserveredfalse
> > 2018-02-09 13:31:48,854 ERROR [c.c.v.VmWorkJobHandlerProxy]
> > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > (logid:0afb959d) Invocation exception, caused by:
> > com.cloud.exception.InsufficientServerCapacityException: Unable to
> create
> > a
> > deployment for VM[User|i-2-11-VM]Scope=interface
> com.cloud.dc.DataCenter;
> > id=1
> > 2018-02-09 13:31:48,854 INFO  [c.c.v.VmWorkJobHandlerProxy]
> > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896 ctx-9ffd0cf1)
> > (logid:0afb959d) Rethrow exception
> > com.cloud.exception.InsufficientServerCapacityException: Unable to
> create
> > a
> > deployment for VM[User|i-2-11-VM]Scope=interface
> com.cloud.dc.DataCenter;
> > id=1
> > 2018-02-09 13:31:48,854 DEBUG [c.c.v.VmWorkJobDispatcher]
> > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896) (logid:0afb959d) Done
> > with run of VM work job: com.cloud.vm.VmWorkStart for VM 11, job origin:
> > 893
> > 2018-02-09 13:31:48,854 ERROR [c.c.v.VmWorkJobDispatcher]
> > (Work-Job-Executor-7:ctx-a92d467b job-893/job-896) (logid:0afb959d)
> Unable
> > to complete AsyncJobVO {id:896, userId: 2, accountId: 2, instanceType:
> > null, inst

Re: cloudstack-management fails to start after upgrade 4.10 -> 4.11

[Jevgeni Zolotarjov]

(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.cloud.exception.InsufficientServerCapacityException: Unable
to create a deployment for VM[User|i-2-11-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1072)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4927)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
... 18 more


On Fri, Feb 9, 2018 at 3:05 PM, Jevgeni Zolotarjov <j.zolotar...@gmail.com>
wrote:

> OK, I pushed through and got management server running.
> The steps I made:
> * recovered DB from backup
> * downgraded everything 4.11->4.10
> * installed systemvmtemplates
> * upgraded 4.10 -> 4.11
>
> THEN!!! To make it run I had to go through all manual DB altering.
> Meaning, there is definitely something wrong in shapeblue packages. I could
> not locate the necessary DB scripts in the package. So I fetched an update
> from github repo.
>
> Even then it failed to start and I had to do some more DB altering from
> file https://github.com/apache/cloudstack/blob/4.11/
> engine/schema/resources/META-INF/db/schema-4930to41000.sql
> Namely:
> ALTER TABLE `cloud`.`sslcerts` ADD COLUMN `name` varchar(255) NULL default
> NULL COMMENT 'Name of the Certificate';
> ALTER TABLE `cloud`.`network_offerings` ADD COLUMN `service_package_id`
> varchar(255) NULL default NULL COMMENT 'Netscaler ControlCenter Service
> Package';
>
> OK. Here I am.
>
> BUT! Not all my VMs are starting. On attempt to start them, manually, it
> says
>
> "Unable to start a VM due to insufficient capacity"
>
> OK, I am having VMs taking more cores, than actual physical number of
> cores.
> VM cores - 35,
> physical cores - 24
>
> It was not the problem with previous version 4.10 - and it was correct.  I
> want to decide myself if I want to allow cpu overprovisioning.
>
> I consider this a critical bug.
>
> On Fri, Feb 9, 2018 at 11:51 AM, Jevgeni Zolotarjov <
> j.zolotar...@gmail.com> wrote:
>
>> my host in Centos7. I upgraded just by changing the URL in repository and
>> yum update
>>
>> How do I upgrade systemvm templates now?
>>
>> On Fri, Feb 9, 2018 at 11:49 AM, Daan Hoogland <daan.hoogl...@gmail.com>
>> wrote:
>>
>>> ok, this is also an upgrade issue. If all is well you installed them
>>> first.
>>> You must have downloaded the new systemvm templates and installed them as
>>> per the upgrade instructions (didn't read them myself this time so i'd
>>> have
>>> to check) and the upgrade will then mark them as the current templates
>>> for
>>> systemVMs.
>>> This error indicates that something in that process went wrong.
>>>
>>> On Fri, Feb 9, 2018 at 10:44 AM, Jevgeni Zolotarjov <
>>> j.zolotar...@gmail.com>
>>> wrote:
>>>
>>> > I have a backup and noone is accessing the installation during upgrade
>>> > process
>>> >
>>> > Now:
>>> > I moved 1 bit further.
>>> >
>>> > Now I am having this error
>>> > 2018-02-09 09:38:05,475 DEBUG [c.c.u.d.ScriptRunner] (main:null)
>>> (logid:)
>>> > ALTER TABLE cloud.ldap_configuration ADD COLUMN domain_id BIGINT(20)
>>> > DEFAULT NULL
>>> > 2018-02-09 09:38:05,476 DEBUG [c.c.u.d.ScriptRunner] (main:null)
>>> (logid:)
>>> > ALTER TABLE cloud.ldap_trust_map ADD COLUMN account_id BIGINT(20)
>>> DEFAULT 0
>>> > 2018-02-09 09:38:05,478 DEBUG [c.c.u.d.ScriptRunner] (main:null)
>>> (logid:)
>>> > ALTER TABLE cloud.ldap_trust_map DROP FOREIGN KEY
>>> > fk_ldap_trust_map__domain_id
>>> > 2018-02-09 09:38:05,481 DEBUG [c.c.u.d.ScriptRunner] (main:null)
>>> (logid:)
>>> > DROP INDEX uk_ldap_trust_map__domain_id ON cloud.ldap_trust_map
>>> > 2018-02-09 09:38:05,482 DEBUG [c.c.u.d.ScriptRunner] (main:null)
>>> (logid:)
>>> > CREATE UNIQUE INDEX uk_ldap_trust_map__bind_location ON ldap_trust_map
>>> > (domain_id, account_id)
>>> > 2018-02-09 09:38:05,48

Re: cloudstack-management fails to start after upgrade 4.10 -> 4.11

[Jevgeni Zolotarjov]

OK, I pushed through and got management server running.
The steps I made:
* recovered DB from backup
* downgraded everything 4.11->4.10
* installed systemvmtemplates
* upgraded 4.10 -> 4.11

THEN!!! To make it run I had to go through all manual DB altering. Meaning,
there is definitely something wrong in shapeblue packages. I could not
locate the necessary DB scripts in the package. So I fetched an update from
github repo.

Even then it failed to start and I had to do some more DB altering from
file
https://github.com/apache/cloudstack/blob/4.11/engine/schema/resources/META-INF/db/schema-4930to41000.sql
Namely:
ALTER TABLE `cloud`.`sslcerts` ADD COLUMN `name` varchar(255) NULL default
NULL COMMENT 'Name of the Certificate';
ALTER TABLE `cloud`.`network_offerings` ADD COLUMN `service_package_id`
varchar(255) NULL default NULL COMMENT 'Netscaler ControlCenter Service
Package';

OK. Here I am.

BUT! Not all my VMs are starting. On attempt to start them, manually, it
says

"Unable to start a VM due to insufficient capacity"

OK, I am having VMs taking more cores, than actual physical number of
cores.
VM cores - 35,
physical cores - 24

It was not the problem with previous version 4.10 - and it was correct.  I
want to decide myself if I want to allow cpu overprovisioning.

I consider this a critical bug.

On Fri, Feb 9, 2018 at 11:51 AM, Jevgeni Zolotarjov <j.zolotar...@gmail.com>
wrote:

> my host in Centos7. I upgraded just by changing the URL in repository and
> yum update
>
> How do I upgrade systemvm templates now?
>
> On Fri, Feb 9, 2018 at 11:49 AM, Daan Hoogland <daan.hoogl...@gmail.com>
> wrote:
>
>> ok, this is also an upgrade issue. If all is well you installed them
>> first.
>> You must have downloaded the new systemvm templates and installed them as
>> per the upgrade instructions (didn't read them myself this time so i'd
>> have
>> to check) and the upgrade will then mark them as the current templates for
>> systemVMs.
>> This error indicates that something in that process went wrong.
>>
>> On Fri, Feb 9, 2018 at 10:44 AM, Jevgeni Zolotarjov <
>> j.zolotar...@gmail.com>
>> wrote:
>>
>> > I have a backup and noone is accessing the installation during upgrade
>> > process
>> >
>> > Now:
>> > I moved 1 bit further.
>> >
>> > Now I am having this error
>> > 2018-02-09 09:38:05,475 DEBUG [c.c.u.d.ScriptRunner] (main:null)
>> (logid:)
>> > ALTER TABLE cloud.ldap_configuration ADD COLUMN domain_id BIGINT(20)
>> > DEFAULT NULL
>> > 2018-02-09 09:38:05,476 DEBUG [c.c.u.d.ScriptRunner] (main:null)
>> (logid:)
>> > ALTER TABLE cloud.ldap_trust_map ADD COLUMN account_id BIGINT(20)
>> DEFAULT 0
>> > 2018-02-09 09:38:05,478 DEBUG [c.c.u.d.ScriptRunner] (main:null)
>> (logid:)
>> > ALTER TABLE cloud.ldap_trust_map DROP FOREIGN KEY
>> > fk_ldap_trust_map__domain_id
>> > 2018-02-09 09:38:05,481 DEBUG [c.c.u.d.ScriptRunner] (main:null)
>> (logid:)
>> > DROP INDEX uk_ldap_trust_map__domain_id ON cloud.ldap_trust_map
>> > 2018-02-09 09:38:05,482 DEBUG [c.c.u.d.ScriptRunner] (main:null)
>> (logid:)
>> > CREATE UNIQUE INDEX uk_ldap_trust_map__bind_location ON ldap_trust_map
>> > (domain_id, account_id)
>> > 2018-02-09 09:38:05,488 ERROR [c.c.u.PropertiesUtil] (main:null)
>> (logid:)
>> > Unable to find properties file: commands.properties
>> > 2018-02-09 09:38:05,489 DEBUG [c.c.u.d.Upgrade41000to41100] (main:null)
>> > (logid:) No commands.properties file was found, enabling dynamic roles
>> by
>> > setting dynamic.apichecker.enabled to true if not already enabled.
>> > 2018-02-09 09:38:05,490 DEBUG [c.c.u.d.Upgrade41000to41100] (main:null)
>> > (logid:) Done validating base64 content of user data
>> > 2018-02-09 09:38:05,490 DEBUG [c.c.u.d.Upgrade41000to41100] (main:null)
>> > (logid:) Updating System Vm template IDs
>> > 2018-02-09 09:38:05,493 DEBUG [c.c.u.d.Upgrade41000to41100] (main:null)
>> > (logid:) Updating LXC System Vms
>> > 2018-02-09 09:38:05,493 WARN  [c.c.u.d.Upgrade41000to41100] (main:null)
>> > (logid:) 4.11.0.0LXC SystemVm template not found. LXC hypervisor is not
>> > used, so not failing upgrade
>> > 2018-02-09 09:38:05,494 DEBUG [c.c.u.d.Upgrade41000to41100] (main:null)
>> > (logid:) Updating Hyperv System Vms
>> > 2018-02-09 09:38:05,495 WARN  [c.c.u.d.Upgrade41000to41100] (main:null)
>> > (logid:) 4.11.0.0Hyperv SystemVm template not found. Hyperv hypervisor
>> is
>> > not used, so not failing upgrade
>> > 2018-02-09 09:38:05,496 DEBUG [c.c.u.d.Upgrade410

Re: cloudstack-management fails to start after upgrade 4.10 -> 4.11

[Jevgeni Zolotarjov]

my host in Centos7. I upgraded just by changing the URL in repository and
yum update

How do I upgrade systemvm templates now?

On Fri, Feb 9, 2018 at 11:49 AM, Daan Hoogland <daan.hoogl...@gmail.com>
wrote:

> ok, this is also an upgrade issue. If all is well you installed them first.
> You must have downloaded the new systemvm templates and installed them as
> per the upgrade instructions (didn't read them myself this time so i'd have
> to check) and the upgrade will then mark them as the current templates for
> systemVMs.
> This error indicates that something in that process went wrong.
>
> On Fri, Feb 9, 2018 at 10:44 AM, Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> wrote:
>
> > I have a backup and noone is accessing the installation during upgrade
> > process
> >
> > Now:
> > I moved 1 bit further.
> >
> > Now I am having this error
> > 2018-02-09 09:38:05,475 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> > ALTER TABLE cloud.ldap_configuration ADD COLUMN domain_id BIGINT(20)
> > DEFAULT NULL
> > 2018-02-09 09:38:05,476 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> > ALTER TABLE cloud.ldap_trust_map ADD COLUMN account_id BIGINT(20)
> DEFAULT 0
> > 2018-02-09 09:38:05,478 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> > ALTER TABLE cloud.ldap_trust_map DROP FOREIGN KEY
> > fk_ldap_trust_map__domain_id
> > 2018-02-09 09:38:05,481 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> > DROP INDEX uk_ldap_trust_map__domain_id ON cloud.ldap_trust_map
> > 2018-02-09 09:38:05,482 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> > CREATE UNIQUE INDEX uk_ldap_trust_map__bind_location ON ldap_trust_map
> > (domain_id, account_id)
> > 2018-02-09 09:38:05,488 ERROR [c.c.u.PropertiesUtil] (main:null) (logid:)
> > Unable to find properties file: commands.properties
> > 2018-02-09 09:38:05,489 DEBUG [c.c.u.d.Upgrade41000to41100] (main:null)
> > (logid:) No commands.properties file was found, enabling dynamic roles by
> > setting dynamic.apichecker.enabled to true if not already enabled.
> > 2018-02-09 09:38:05,490 DEBUG [c.c.u.d.Upgrade41000to41100] (main:null)
> > (logid:) Done validating base64 content of user data
> > 2018-02-09 09:38:05,490 DEBUG [c.c.u.d.Upgrade41000to41100] (main:null)
> > (logid:) Updating System Vm template IDs
> > 2018-02-09 09:38:05,493 DEBUG [c.c.u.d.Upgrade41000to41100] (main:null)
> > (logid:) Updating LXC System Vms
> > 2018-02-09 09:38:05,493 WARN  [c.c.u.d.Upgrade41000to41100] (main:null)
> > (logid:) 4.11.0.0LXC SystemVm template not found. LXC hypervisor is not
> > used, so not failing upgrade
> > 2018-02-09 09:38:05,494 DEBUG [c.c.u.d.Upgrade41000to41100] (main:null)
> > (logid:) Updating Hyperv System Vms
> > 2018-02-09 09:38:05,495 WARN  [c.c.u.d.Upgrade41000to41100] (main:null)
> > (logid:) 4.11.0.0Hyperv SystemVm template not found. Hyperv hypervisor is
> > not used, so not failing upgrade
> > 2018-02-09 09:38:05,496 DEBUG [c.c.u.d.Upgrade41000to41100] (main:null)
> > (logid:) Updating KVM System Vms
> > 2018-02-09 09:38:05,497 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null)
> > (logid:) Unable to upgrade the database
> > com.cloud.utils.exception.CloudRuntimeException: 4.11.0.0KVM SystemVm
> > template not found. Cannot upgrade system Vms
> > at
> > com.cloud.upgrade.dao.Upgrade41000to41100.updateSystemVmTemplates(
> > Upgrade41000to41100.java:270)
> > at
> > com.cloud.upgrade.dao.Upgrade41000to41100.performDataMigration(
> > Upgrade41000to41100.java:71)
> > at
> > com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(
> > DatabaseUpgradeChecker.java:561)
> > at
> > com.cloud.upgrade.DatabaseUpgradeChecker.check(
> > DatabaseUpgradeChecker.java:641)
> > at
> > org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.
> > checkIntegrity(CloudStackExtendedLifeCycle.java:65)
> > at
> > org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.
> start(
> > CloudStackExtendedLifeCycle.java:55)
> > at
> > org.springframework.context.support.DefaultLifecycleProcessor.doStart(
> > DefaultLifecycleProcessor.java:183)
> > at
> > org.springframework.context.support.DefaultLifecycleProcessor.
> access$200(
> > DefaultLifecycleProcessor.java:52)
> > at
> > org.springframework.context.support.DefaultLifecycleProcessor$
> > LifecycleGroup.start(DefaultLifecycleProcessor.java:358)
> > at
> > org.springframework.context.support.DefaultLifecycleProcessor.
> startBeans(
> &

Re: cloudstack-management fails to start after upgrade 4.10 -> 4.11

[Jevgeni Zolotarjov]

)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:71)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.(CloudStackSpringContext.java:58)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.(CloudStackSpringContext.java:62)
at
org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52)
at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:890)
at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532)
at
org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:853)
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344)
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1515)
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1477)
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785)



On Fri, Feb 9, 2018 at 11:42 AM, Daan Hoogland <daan.hoogl...@gmail.com>
wrote:

> before you continue Jevgeni,
>
> Do you have a good backup and is no-one able to access this installation
> with UI or API, when you roll back?
>
> On Fri, Feb 9, 2018 at 10:15 AM, Ernie Janse van Rensburg <
> ernie.jvrensb...@shapeblue.com> wrote:
>
> > engine/schema/resources/META-INF/db/schema-41000to41100.sql:
> >
> > ____
> > From: Jevgeni Zolotarjov <j.zolotar...@gmail.com>
> > Sent: Friday, February 9, 2018 10:59:27 AM
> > To: users@cloudstack.apache.org
> > Subject: Re: cloudstack-management fails to start after upgrade 4.10 ->
> > 4.11
> >
> > I dropped the column. But then another error like this appeared
> >
> > Please advise, where are the DB update script located? so I can manually
> > inspect that.
> >
> > On Fri, Feb 9, 2018 at 10:29 AM, Ernie Janse van Rensburg <
> > ernie.jvrensb...@shapeblue.com> wrote:
> >
> > > Hi Jevgeni
> > >
> > >
> > > It looks like there was a database error during the upgrade process.
> > >
> > >
> > > A SQL script that is trying to add a column 'for_vpc' to the TABLE
> > > 'cloud.network_offerings' but the column already exists, for some
> reason,
> > > so it fails because mysql does not allow 2 columns with the same name
> in
> > > the same table.
> > >
> > >
> > > The column might have already existed from a previous upgrade or from
> > > running the 4.11 upgrade more than once, and perhaps the SQL script is
> > not
> > > idempotent.
> > >
> > >
> > > I suggest to manually drop the column on the table and run the 4.11
> > > upgrade process again.
> > >
> > >
> > > Regards
> > >
> > >
> > > Ernie
> > >
> > > 
> > > From: Jevgeni Zolotarjov <j.zolotar...@gmail.com>
> > > Sent: Friday, February 9, 2018 10:10:59 AM
> > > To: users@cloudstack.apache.org
> > > Subject: cloudstack-management fails to start after upgrade 4.10 ->
> 4.11
> > >
> > > cloudstack-management fails to start after upgrade from 4.10 to 4.11
> > >
> > > management-server.log:
> > > 2018-02-09 07:49:50,842 DEBUG [c.c.u.DatabaseUpgradeChecker]
> (main:null)
> > > (logid:) Running upgrade Upgrade41000to41100 to upgrade from
> > > 4.10.0.0-4.11.0.0 to 4.11.0.0
> > > 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null)
> (logid:)
> > > -- Licensed to the Apache Software Foundation (ASF) under one
> > > 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null)
> (logid:)
> > > -- or more contributor license agreements.  See the NOTICE file
> > > 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null)
> (logid:)
> > > -- distributed with this work for additional information
> > > 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null)
> (logid:)
> > > -- regarding copyright ownership.  The ASF licenses this file
> > > 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null)
> (logid:)
> > > -- to you under the Apache License, Version 2.0 (the
> > > 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null)
> (logid:)
> > > -- "License"); you may not use this file except in compliance
> > > 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null)
> (logid:)
> > 

Re: cloudstack-management fails to start after upgrade 4.10 -> 4.11

[Jevgeni Zolotarjov]

I dropped the column. But then another error like this appeared

Please advise, where are the DB update script located? so I can manually
inspect that.

On Fri, Feb 9, 2018 at 10:29 AM, Ernie Janse van Rensburg <
ernie.jvrensb...@shapeblue.com> wrote:

> Hi Jevgeni
>
>
> It looks like there was a database error during the upgrade process.
>
>
> A SQL script that is trying to add a column 'for_vpc' to the TABLE
> 'cloud.network_offerings' but the column already exists, for some reason,
> so it fails because mysql does not allow 2 columns with the same name in
> the same table.
>
>
> The column might have already existed from a previous upgrade or from
> running the 4.11 upgrade more than once, and perhaps the SQL script is not
> idempotent.
>
>
> I suggest to manually drop the column on the table and run the 4.11
> upgrade process again.
>
>
> Regards
>
>
> Ernie
>
> 
> From: Jevgeni Zolotarjov <j.zolotar...@gmail.com>
> Sent: Friday, February 9, 2018 10:10:59 AM
> To: users@cloudstack.apache.org
> Subject: cloudstack-management fails to start after upgrade 4.10 -> 4.11
>
> cloudstack-management fails to start after upgrade from 4.10 to 4.11
>
> management-server.log:
> 2018-02-09 07:49:50,842 DEBUG [c.c.u.DatabaseUpgradeChecker] (main:null)
> (logid:) Running upgrade Upgrade41000to41100 to upgrade from
> 4.10.0.0-4.11.0.0 to 4.11.0.0
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- Licensed to the Apache Software Foundation (ASF) under one
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- or more contributor license agreements.  See the NOTICE file
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- distributed with this work for additional information
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- regarding copyright ownership.  The ASF licenses this file
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- to you under the Apache License, Version 2.0 (the
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- "License"); you may not use this file except in compliance
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- with the License.  You may obtain a copy of the License at
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> --
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> --   http://www.apache.org/licenses/LICENSE-2.0
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> --
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- Unless required by applicable law or agreed to in writing,
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- software distributed under the License is distributed on an
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- KIND, either express or implied.  See the License for the
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- specific language governing permissions and limitations
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- under the License.
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> --;
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- Schema upgrade from 4.10.0.0 to 4.11.0.0
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> --;
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> -- Add For VPC flag
> 2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
> ALTER TABLE cloud.network_offerings ADD COLUMN for_vpc INT(1) NOT NULL
> DEFAULT 0
> 2018-02-09 07:49:50,847 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:)
> Error executing: ALTER TABLE cloud.network_offerings ADD COLUMN for_vpc
> INT(1) NOT NULL DEFAULT 0
> 2018-02-09 07:49:50,848 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:)
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate
> column
> name 'for_vpc'
> 2018-02-09 07:49:50,849 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null)
> (logid:) Unable to execute upgrade script
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate
> column
> name 'for_vpc'
> at com.cloud.utils.db.ScriptRunner.runScript(
> ScriptRunner.java:185)
> at com.

cloudstack-management fails to start after upgrade 4.10 -> 4.11

[Jevgeni Zolotarjov]

cloudstack-management fails to start after upgrade from 4.10 to 4.11

management-server.log:
2018-02-09 07:49:50,842 DEBUG [c.c.u.DatabaseUpgradeChecker] (main:null)
(logid:) Running upgrade Upgrade41000to41100 to upgrade from
4.10.0.0-4.11.0.0 to 4.11.0.0
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
-- Licensed to the Apache Software Foundation (ASF) under one
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
-- or more contributor license agreements.  See the NOTICE file
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
-- distributed with this work for additional information
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
-- regarding copyright ownership.  The ASF licenses this file
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
-- to you under the Apache License, Version 2.0 (the
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
-- "License"); you may not use this file except in compliance
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
-- with the License.  You may obtain a copy of the License at
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) --
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
--   http://www.apache.org/licenses/LICENSE-2.0
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:) --
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
-- Unless required by applicable law or agreed to in writing,
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
-- software distributed under the License is distributed on an
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
-- KIND, either express or implied.  See the License for the
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
-- specific language governing permissions and limitations
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
-- under the License.
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
--;
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
-- Schema upgrade from 4.10.0.0 to 4.11.0.0
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
--;
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
-- Add For VPC flag
2018-02-09 07:49:50,846 DEBUG [c.c.u.d.ScriptRunner] (main:null) (logid:)
ALTER TABLE cloud.network_offerings ADD COLUMN for_vpc INT(1) NOT NULL
DEFAULT 0
2018-02-09 07:49:50,847 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:)
Error executing: ALTER TABLE cloud.network_offerings ADD COLUMN for_vpc
INT(1) NOT NULL DEFAULT 0
2018-02-09 07:49:50,848 ERROR [c.c.u.d.ScriptRunner] (main:null) (logid:)
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column
name 'for_vpc'
2018-02-09 07:49:50,849 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null)
(logid:) Unable to execute upgrade script
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column
name 'for_vpc'
at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:185)
at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:87)
at
com.cloud.upgrade.DatabaseUpgradeChecker.runScript(DatabaseUpgradeChecker.java:459)
at
com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:557)
at
com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:641)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:65)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:55)
at
org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:183)
at
org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:52)
at
org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:358)
at
org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:159)
at
org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123)
at
org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:884)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:552)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:145)
at

Re: Unable to create a deployment for VM

[Jevgeni Zolotarjov]

true, it does.

But for understanding the problem, I would have to
1. analyze the logs,
2. multiply [service offering core count] x [service offering MHz] and
3. check it against the remainder from the indicator on the dashboard.
Yes, it works that way too. But then I also have to
4. consider cpu.overprovisioning.factor (as I learned from you)

Well, I guess everybody would expect to see meaningful error message from
mature product. So indeed, I would like to contribute and at least post
this feature request.
Does someone know, how that works?

Best regards,
Jevgeni

On Fri, Oct 20, 2017 at 4:14 PM, Dag Sonstebo <dag.sonst...@shapeblue.com>
wrote:

> Agree the error could be clearer – but if you check your CloudStack
> dashboard the pie chart for CPU already tells you your CPU utilisation.
>
> You can also use the listCapacity API call mentioned before:
>
> # cloudmonkey list capacity
> count = 9
> capacity:
> +--+--+-
> ++--+---+
> | capacityused |zoneid| percentused |
> capacitytotal  | type |  zonename |
> +--+--+-
> ++--+---+
> | 6792 | d4b9d32e-d779-48b8-814d-d7847d55a684 |31.51|
>  21552  |  1   zonenamehere |
> | 332262574592 | d4b9d32e-d779-48b8-814d-d7847d55a684 | 1.11|
> 30056223080448 |  3   | zonenamehere |
> |  17  | d4b9d32e-d779-48b8-814d-d7847d55a684 |  85 |
>20   |  4   | zonenamehere |
> |  2   | d4b9d32e-d779-48b8-814d-d7847d55a684 |  10 |
>20   |  5   zonenamehere |
> |  6   | d4b9d32e-d779-48b8-814d-d7847d55a684 |  30 |
>20   |  7   | zonenamehere |
> |  5234491392  | d4b9d32e-d779-48b8-814d-d7847d55a684 |35.24|
> 14854279424   |  0   | zonenamehere |
> | 261059772416 | d4b9d32e-d779-48b8-814d-d7847d55a684 | 3.47|
> 7514055770112  |  6   | zonenamehere |
> | 522119544832 | d4b9d32e-d779-48b8-814d-d7847d55a684 | 3.47|
> 15028111540224 |  2   | zonenamehere |
> |  0   | d4b9d32e-d779-48b8-814d-d7847d55a684 |  0  |
>0|  19  | zonenamehere |
> +--+--+-
> ++--+-----------+
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 20/10/2017, 13:33, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com> wrote:
>
> Thanks a lot!
>
> It was it!
>
> Actually the error reporting could be done in a wiser way to inform
> administrator about what is the problem in particular. Do you know,
> where
> can I request this feature?
>
> Regards,
> Jevgeni
>
> On Fri, Oct 20, 2017 at 1:25 PM, Dag Sonstebo <
> dag.sonst...@shapeblue.com>
> wrote:
>
> > Capacity type 1 is CPU: http://cloudstack.apache.org/
> > api/apidocs-4.10/apis/listCapacity.html
> >
> > You would look at changing cluster.cpu.allocated.
> capacity.disablethreshold
> > in global settings – or the same under your cluster settings.
> > You can also play with cpu.overprovisioning.factor – but keep in mind
>     > overprovisioning is a science in itself – which has lots of gotcha’s
> > depending on your workload.
> >
> > Regards,
> > Dag Sonstebo
> > Cloud Architect
> > ShapeBlue
> >
> > On 20/10/2017, 11:16, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com>
> wrote:
> >
> > I changed cluster.threshold.enabled to false. But it didn't help.
> > I guess the root cause it something different.
> >
> > My storage capacity if above 3TB, and only under 300GB used.
>     >
> > No threshold setting was changed ever since installation.
> >
> > On Fri, Oct 20, 2017 at 12:59 PM, Ivan Kudryavtsev <
> > kudryavtsev...@bw-sw.com
> > > wrote:
> >
> > > Just look for threshold. Logs don't include such information.
> > >
> > > 2017-10-20 16:56 GMT+07:00 Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com>:
> > >
> > > > What is exact parameter name?
> > > >
> > > > On Fri, Oct 20, 2017 at 12:44 PM, Ivan Kudryavtsev <
> > > > kudryavtsev...@bw-sw.com
> > > > > wrote:
> > > >
> > > > > Ah, OK
> &

Re: Unable to create a deployment for VM

[Jevgeni Zolotarjov]

Thanks a lot!

It was it!

Actually the error reporting could be done in a wiser way to inform
administrator about what is the problem in particular. Do you know, where
can I request this feature?

Regards,
Jevgeni

On Fri, Oct 20, 2017 at 1:25 PM, Dag Sonstebo <dag.sonst...@shapeblue.com>
wrote:

> Capacity type 1 is CPU: http://cloudstack.apache.org/
> api/apidocs-4.10/apis/listCapacity.html
>
> You would look at changing cluster.cpu.allocated.capacity.disablethreshold
> in global settings – or the same under your cluster settings.
> You can also play with cpu.overprovisioning.factor – but keep in mind
> overprovisioning is a science in itself – which has lots of gotcha’s
> depending on your workload.
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 20/10/2017, 11:16, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com> wrote:
>
> I changed cluster.threshold.enabled to false. But it didn't help.
> I guess the root cause it something different.
>
> My storage capacity if above 3TB, and only under 300GB used.
>
> No threshold setting was changed ever since installation.
>
> On Fri, Oct 20, 2017 at 12:59 PM, Ivan Kudryavtsev <
> kudryavtsev...@bw-sw.com
> > wrote:
>
> > Just look for threshold. Logs don't include such information.
> >
> > 2017-10-20 16:56 GMT+07:00 Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>:
> >
> > > What is exact parameter name?
> > >
> > > On Fri, Oct 20, 2017 at 12:44 PM, Ivan Kudryavtsev <
> > > kudryavtsev...@bw-sw.com
> > > > wrote:
> > >
> > > > Ah, OK
> > > >
> > > > 2017-10-20 09:10:53,697 DEBUG [c.c.d.FirstFitPlanner]
> > > > (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c)
> > (logid:25e762a0)
> > > > Cannot allocate cluster list [1] for vm creation since their
> allocated
> > > > percentage crosses the disable capacity threshold defined at each
> > > cluster/
> > > > at global value for capacity Type : 1, skipping these clusters
> > > >
> > > > That's it. You have to change global variables or cluster
> configuration
> > > > variables, because you're close to thresholds.
> > > >
> > > > 2017-10-20 16:40 GMT+07:00 Jevgeni Zolotarjov <
> j.zolotar...@gmail.com
> > >:
> > > >
> > > > > Hi,
> > > > >
> > > > > Yes I have one host. But I have exactly zero affinity groups.
> I could
> > > > > successfully add few instances and they are running.
> > > > > The
> > > > >
> > > > >
> > > > > On Fri, Oct 20, 2017 at 12:35 PM, Ivan Kudryavtsev <
> > > > > kudryavtsev...@bw-sw.com
> > > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > It seems you have one host and place several VMs in a single
> > > > > anti-affinity
> > > > > > group, don't you? This is how it should really work if you
> do that
> > > > thing.
> > > > > > Anti-affinity groups are meaningless if you don't have
> several
> > hosts.
> > > > > >
> > > > > > 2017-10-20 16:19 GMT+07:00 Jevgeni Zolotarjov <
> > > j.zolotar...@gmail.com
> > > > >:
> > > > > >
> > > > > > > I could successfully create and run an instance with
> prepared
> > > Compute
> > > > > > > offering. But when I try to add one more instance of the
> same
> > > Compute
> > > > > > > offering, I am getting an error.
> > > > > > > Unable to create a deployment for VM
> > > > > > >
> > > > > > > There are plenty of resources on host: cores, RAM, storage.
> > > > > > >
> > > > > > > Pleas help
> > > > > > >
> > > > > > > management-server.log:
> > > > > > > 2017-10-20 09:10:43,713 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > > > > (AgentManager-Handler-3:null) (logid:) SeqA 5-377872:
> Processing
> > > Seq
> > > > > > > 5-377872:  { Cmd , MgmtId: -1, via: 5, Ver: v1, Flags: 11,
> > > >

Re: Unable to create a deployment for VM

[Jevgeni Zolotarjov]

I changed cluster.threshold.enabled to false. But it didn't help.
I guess the root cause it something different.

My storage capacity if above 3TB, and only under 300GB used.

No threshold setting was changed ever since installation.

On Fri, Oct 20, 2017 at 12:59 PM, Ivan Kudryavtsev <kudryavtsev...@bw-sw.com
> wrote:

> Just look for threshold. Logs don't include such information.
>
> 2017-10-20 16:56 GMT+07:00 Jevgeni Zolotarjov <j.zolotar...@gmail.com>:
>
> > What is exact parameter name?
> >
> > On Fri, Oct 20, 2017 at 12:44 PM, Ivan Kudryavtsev <
> > kudryavtsev...@bw-sw.com
> > > wrote:
> >
> > > Ah, OK
> > >
> > > 2017-10-20 09:10:53,697 DEBUG [c.c.d.FirstFitPlanner]
> > > (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c)
> (logid:25e762a0)
> > > Cannot allocate cluster list [1] for vm creation since their allocated
> > > percentage crosses the disable capacity threshold defined at each
> > cluster/
> > > at global value for capacity Type : 1, skipping these clusters
> > >
> > > That's it. You have to change global variables or cluster configuration
> > > variables, because you're close to thresholds.
> > >
> > > 2017-10-20 16:40 GMT+07:00 Jevgeni Zolotarjov <j.zolotar...@gmail.com
> >:
> > >
> > > > Hi,
> > > >
> > > > Yes I have one host. But I have exactly zero affinity groups. I could
> > > > successfully add few instances and they are running.
> > > > The
> > > >
> > > >
> > > > On Fri, Oct 20, 2017 at 12:35 PM, Ivan Kudryavtsev <
> > > > kudryavtsev...@bw-sw.com
> > > > > wrote:
> > > >
> > > > > Hi,
> > > > > It seems you have one host and place several VMs in a single
> > > > anti-affinity
> > > > > group, don't you? This is how it should really work if you do that
> > > thing.
> > > > > Anti-affinity groups are meaningless if you don't have several
> hosts.
> > > > >
> > > > > 2017-10-20 16:19 GMT+07:00 Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com
> > > >:
> > > > >
> > > > > > I could successfully create and run an instance with prepared
> > Compute
> > > > > > offering. But when I try to add one more instance of the same
> > Compute
> > > > > > offering, I am getting an error.
> > > > > > Unable to create a deployment for VM
> > > > > >
> > > > > > There are plenty of resources on host: cores, RAM, storage.
> > > > > >
> > > > > > Pleas help
> > > > > >
> > > > > > management-server.log:
> > > > > > 2017-10-20 09:10:43,713 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > > > (AgentManager-Handler-3:null) (logid:) SeqA 5-377872: Processing
> > Seq
> > > > > > 5-377872:  { Cmd , MgmtId: -1, via: 5, Ver: v1, Flags: 11,
> > > > > > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand"
> > > > > > :{"_proxyVmId":1,"_loadInfo":"{\n
> > > > > > \"connections\": []\n}","wait":0}}] }
> > > > > > 2017-10-20 09:10:43,715 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > > > (AgentManager-Handler-3:null) (logid:) SeqA 5-377872: Sending Seq
> > > > > > 5-377872:  { Ans: , MgmtId: 264216221068220, via: 5, Ver: v1,
> > Flags:
> > > > > > 100010,
> > > > > > [{"com.cloud.agent.api.AgentControlAnswer":{"result":
> > > true,"wait":0}}]
> > > > }
> > > > > > 2017-10-20 09:10:45,239 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (AsyncJobMgr-Heartbeat-1:ctx-566e7818) (logid:24eee607) Begin
> > > cleanup
> > > > > > expired async-jobs
> > > > > > 2017-10-20 09:10:45,243 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (AsyncJobMgr-Heartbeat-1:ctx-566e7818) (logid:24eee607) End
> > cleanup
> > > > > > expired
> > > > > > async-jobs
> > > > > > 2017-10-20 09:10:53,566 DEBUG [c.c.a.ApiServlet]
> > > > > > (catalina-exec-6:ctx-3911a4cc) (logid:41db964f) ===START===
> > > > > 192.168.1.194
> > > > > > -- GET
> > > > > > command=deployVirtualMachine=json=b3d5b8bb-
> > > > > >

Re: Unable to create a deployment for VM

[Jevgeni Zolotarjov]

What is exact parameter name?

On Fri, Oct 20, 2017 at 12:44 PM, Ivan Kudryavtsev <kudryavtsev...@bw-sw.com
> wrote:

> Ah, OK
>
> 2017-10-20 09:10:53,697 DEBUG [c.c.d.FirstFitPlanner]
> (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c) (logid:25e762a0)
> Cannot allocate cluster list [1] for vm creation since their allocated
> percentage crosses the disable capacity threshold defined at each cluster/
> at global value for capacity Type : 1, skipping these clusters
>
> That's it. You have to change global variables or cluster configuration
> variables, because you're close to thresholds.
>
> 2017-10-20 16:40 GMT+07:00 Jevgeni Zolotarjov <j.zolotar...@gmail.com>:
>
> > Hi,
> >
> > Yes I have one host. But I have exactly zero affinity groups. I could
> > successfully add few instances and they are running.
> > The
> >
> >
> > On Fri, Oct 20, 2017 at 12:35 PM, Ivan Kudryavtsev <
> > kudryavtsev...@bw-sw.com
> > > wrote:
> >
> > > Hi,
> > > It seems you have one host and place several VMs in a single
> > anti-affinity
> > > group, don't you? This is how it should really work if you do that
> thing.
> > > Anti-affinity groups are meaningless if you don't have several hosts.
> > >
> > > 2017-10-20 16:19 GMT+07:00 Jevgeni Zolotarjov <j.zolotar...@gmail.com
> >:
> > >
> > > > I could successfully create and run an instance with prepared Compute
> > > > offering. But when I try to add one more instance of the same Compute
> > > > offering, I am getting an error.
> > > > Unable to create a deployment for VM
> > > >
> > > > There are plenty of resources on host: cores, RAM, storage.
> > > >
> > > > Pleas help
> > > >
> > > > management-server.log:
> > > > 2017-10-20 09:10:43,713 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > (AgentManager-Handler-3:null) (logid:) SeqA 5-377872: Processing Seq
> > > > 5-377872:  { Cmd , MgmtId: -1, via: 5, Ver: v1, Flags: 11,
> > > > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand"
> > > > :{"_proxyVmId":1,"_loadInfo":"{\n
> > > > \"connections\": []\n}","wait":0}}] }
> > > > 2017-10-20 09:10:43,715 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > (AgentManager-Handler-3:null) (logid:) SeqA 5-377872: Sending Seq
> > > > 5-377872:  { Ans: , MgmtId: 264216221068220, via: 5, Ver: v1, Flags:
> > > > 100010,
> > > > [{"com.cloud.agent.api.AgentControlAnswer":{"result":
> true,"wait":0}}]
> > }
> > > > 2017-10-20 09:10:45,239 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (AsyncJobMgr-Heartbeat-1:ctx-566e7818) (logid:24eee607) Begin
> cleanup
> > > > expired async-jobs
> > > > 2017-10-20 09:10:45,243 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (AsyncJobMgr-Heartbeat-1:ctx-566e7818) (logid:24eee607) End cleanup
> > > > expired
> > > > async-jobs
> > > > 2017-10-20 09:10:53,566 DEBUG [c.c.a.ApiServlet]
> > > > (catalina-exec-6:ctx-3911a4cc) (logid:41db964f) ===START===
> > > 192.168.1.194
> > > > -- GET
> > > > command=deployVirtualMachine=json=b3d5b8bb-
> > > > 2dea-478b-8077-fc813b90d4c6=244d6029-0f9a-4479-
> > > > b11b-09e4fe6c40e1=KVM=
> > > > e0ef2da9-a489-4c42-ab6d-c49fd41b0fd2=
> > > > f39fb02b-4fc0-4f2e-9e4e-dd54f7fc9508=50&
> > securitygroupids=1586621a-
> > > > 8e8d-11e7-8f52-f04da2002bbe=mtl-dev04-app02&
> > > > name=mtl-dev04-app02=us&_=1508490653550
> > > > 2017-10-20 09:10:53,581 DEBUG [c.c.u.AccountManagerImpl]
> > > > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Access
> > > granted
> > > > to Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin] to
> > > > org.apache.cloudstack.quota.vo.ServiceOfferingVO$$
> > > > EnhancerByCGLIB$$4d79c9fa@540d8fcb
> > > > by AffinityGroupAccessChecker
> > > > 2017-10-20 09:10:53,581 DEBUG [c.c.u.AccountManagerImpl]
> > > > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Access
> > > granted
> > > > to Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin] to
> > > > com.cloud.storage.DiskOfferingVO$$EnhancerByCGLIB$$f64346ed@7d0dcb96
> > by
> > > > AffinityGroupAccessChecker
> > > > 2017-10-20 09:10:53,605 DEBUG [c.c.v.UserVmManagerImpl]
> > > > (catalina-exec-6:ctx-391

Re: Unable to create a deployment for VM

[Jevgeni Zolotarjov]

Hi,

Yes I have one host. But I have exactly zero affinity group
s - I dont use affinity groups. I could successfully add few instances and
they are running.
The exceptions says: InsufficientServerCapacityException
But there is plenty of storage.

Disk Total 3.24 TB
Disk Allocated 265.33 GB

On Fri, Oct 20, 2017 at 12:35 PM, Ivan Kudryavtsev <kudryavtsev...@bw-sw.com
> wrote:

> Hi,
> It seems you have one host and place several VMs in a single anti-affinity
> group, don't you? This is how it should really work if you do that thing.
> Anti-affinity groups are meaningless if you don't have several hosts.
>
> 2017-10-20 16:19 GMT+07:00 Jevgeni Zolotarjov <j.zolotar...@gmail.com>:
>
> > I could successfully create and run an instance with prepared Compute
> > offering. But when I try to add one more instance of the same Compute
> > offering, I am getting an error.
> > Unable to create a deployment for VM
> >
> > There are plenty of resources on host: cores, RAM, storage.
> >
> > Pleas help
> >
> > management-server.log:
> > 2017-10-20 09:10:43,713 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentManager-Handler-3:null) (logid:) SeqA 5-377872: Processing Seq
> > 5-377872:  { Cmd , MgmtId: -1, via: 5, Ver: v1, Flags: 11,
> > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand"
> > :{"_proxyVmId":1,"_loadInfo":"{\n
> > \"connections\": []\n}","wait":0}}] }
> > 2017-10-20 09:10:43,715 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentManager-Handler-3:null) (logid:) SeqA 5-377872: Sending Seq
> > 5-377872:  { Ans: , MgmtId: 264216221068220, via: 5, Ver: v1, Flags:
> > 100010,
> > [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> > 2017-10-20 09:10:45,239 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-566e7818) (logid:24eee607) Begin cleanup
> > expired async-jobs
> > 2017-10-20 09:10:45,243 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-566e7818) (logid:24eee607) End cleanup
> > expired
> > async-jobs
> > 2017-10-20 09:10:53,566 DEBUG [c.c.a.ApiServlet]
> > (catalina-exec-6:ctx-3911a4cc) (logid:41db964f) ===START===
> 192.168.1.194
> > -- GET
> > command=deployVirtualMachine=json=b3d5b8bb-
> > 2dea-478b-8077-fc813b90d4c6=244d6029-0f9a-4479-
> > b11b-09e4fe6c40e1=KVM=
> > e0ef2da9-a489-4c42-ab6d-c49fd41b0fd2=
> > f39fb02b-4fc0-4f2e-9e4e-dd54f7fc9508=50=1586621a-
> > 8e8d-11e7-8f52-f04da2002bbe=mtl-dev04-app02&
> > name=mtl-dev04-app02=us&_=1508490653550
> > 2017-10-20 09:10:53,581 DEBUG [c.c.u.AccountManagerImpl]
> > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Access
> granted
> > to Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin] to
> > org.apache.cloudstack.quota.vo.ServiceOfferingVO$$
> > EnhancerByCGLIB$$4d79c9fa@540d8fcb
> > by AffinityGroupAccessChecker
> > 2017-10-20 09:10:53,581 DEBUG [c.c.u.AccountManagerImpl]
> > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Access
> granted
> > to Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin] to
> > com.cloud.storage.DiskOfferingVO$$EnhancerByCGLIB$$f64346ed@7d0dcb96 by
> > AffinityGroupAccessChecker
> > 2017-10-20 09:10:53,605 DEBUG [c.c.v.UserVmManagerImpl]
> > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Allocating
> in
> > the DB for vm
> > 2017-10-20 09:10:53,613 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Allocating
> > entries for VM: VM[User|i-2-23-VM]
> > 2017-10-20 09:10:53,614 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Allocating
> > nics for VM[User|i-2-23-VM]
> > 2017-10-20 09:10:53,614 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Allocating
> nic
> > for vm VM[User|i-2-23-VM] in network Ntwk[204|Guest|6] with requested
> > profile NicProfile[0-0-null-null-null
> > 2017-10-20 09:10:53,619 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Allocating
> > disks for VM[User|i-2-23-VM]
> > 2017-10-20 09:10:53,626 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Allocation
> > completed for VM: VM[User|i-2-23-VM]
> > 2017-10-20 09:10:53,626 DEBUG [c.c.v.UserVmManagerImpl]
> > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Successfully
> > allocated DB entry for VM[User|i-2-23-VM

Re: Unable to create a deployment for VM

[Jevgeni Zolotarjov]

Hi,

Yes I have one host. But I have exactly zero affinity groups. I could
successfully add few instances and they are running.
The


On Fri, Oct 20, 2017 at 12:35 PM, Ivan Kudryavtsev <kudryavtsev...@bw-sw.com
> wrote:

> Hi,
> It seems you have one host and place several VMs in a single anti-affinity
> group, don't you? This is how it should really work if you do that thing.
> Anti-affinity groups are meaningless if you don't have several hosts.
>
> 2017-10-20 16:19 GMT+07:00 Jevgeni Zolotarjov <j.zolotar...@gmail.com>:
>
> > I could successfully create and run an instance with prepared Compute
> > offering. But when I try to add one more instance of the same Compute
> > offering, I am getting an error.
> > Unable to create a deployment for VM
> >
> > There are plenty of resources on host: cores, RAM, storage.
> >
> > Pleas help
> >
> > management-server.log:
> > 2017-10-20 09:10:43,713 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentManager-Handler-3:null) (logid:) SeqA 5-377872: Processing Seq
> > 5-377872:  { Cmd , MgmtId: -1, via: 5, Ver: v1, Flags: 11,
> > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand"
> > :{"_proxyVmId":1,"_loadInfo":"{\n
> > \"connections\": []\n}","wait":0}}] }
> > 2017-10-20 09:10:43,715 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentManager-Handler-3:null) (logid:) SeqA 5-377872: Sending Seq
> > 5-377872:  { Ans: , MgmtId: 264216221068220, via: 5, Ver: v1, Flags:
> > 100010,
> > [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> > 2017-10-20 09:10:45,239 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-566e7818) (logid:24eee607) Begin cleanup
> > expired async-jobs
> > 2017-10-20 09:10:45,243 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-566e7818) (logid:24eee607) End cleanup
> > expired
> > async-jobs
> > 2017-10-20 09:10:53,566 DEBUG [c.c.a.ApiServlet]
> > (catalina-exec-6:ctx-3911a4cc) (logid:41db964f) ===START===
> 192.168.1.194
> > -- GET
> > command=deployVirtualMachine=json=b3d5b8bb-
> > 2dea-478b-8077-fc813b90d4c6=244d6029-0f9a-4479-
> > b11b-09e4fe6c40e1=KVM=
> > e0ef2da9-a489-4c42-ab6d-c49fd41b0fd2=
> > f39fb02b-4fc0-4f2e-9e4e-dd54f7fc9508=50=1586621a-
> > 8e8d-11e7-8f52-f04da2002bbe=mtl-dev04-app02&
> > name=mtl-dev04-app02=us&_=1508490653550
> > 2017-10-20 09:10:53,581 DEBUG [c.c.u.AccountManagerImpl]
> > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Access
> granted
> > to Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin] to
> > org.apache.cloudstack.quota.vo.ServiceOfferingVO$$
> > EnhancerByCGLIB$$4d79c9fa@540d8fcb
> > by AffinityGroupAccessChecker
> > 2017-10-20 09:10:53,581 DEBUG [c.c.u.AccountManagerImpl]
> > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Access
> granted
> > to Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin] to
> > com.cloud.storage.DiskOfferingVO$$EnhancerByCGLIB$$f64346ed@7d0dcb96 by
> > AffinityGroupAccessChecker
> > 2017-10-20 09:10:53,605 DEBUG [c.c.v.UserVmManagerImpl]
> > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Allocating
> in
> > the DB for vm
> > 2017-10-20 09:10:53,613 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Allocating
> > entries for VM: VM[User|i-2-23-VM]
> > 2017-10-20 09:10:53,614 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Allocating
> > nics for VM[User|i-2-23-VM]
> > 2017-10-20 09:10:53,614 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Allocating
> nic
> > for vm VM[User|i-2-23-VM] in network Ntwk[204|Guest|6] with requested
> > profile NicProfile[0-0-null-null-null
> > 2017-10-20 09:10:53,619 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Allocating
> > disks for VM[User|i-2-23-VM]
> > 2017-10-20 09:10:53,626 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Allocation
> > completed for VM: VM[User|i-2-23-VM]
> > 2017-10-20 09:10:53,626 DEBUG [c.c.v.UserVmManagerImpl]
> > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Successfully
> > allocated DB entry for VM[User|i-2-23-VM]
> > 2017-10-20 09:10:53,653 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
> > (API-Job-Executor-40:ctx-a4cff151 job-181) (logid:0c9b9886) Add job-181
> > in

Unable to create a deployment for VM

[Jevgeni Zolotarjov]

I could successfully create and run an instance with prepared Compute
offering. But when I try to add one more instance of the same Compute
offering, I am getting an error.
Unable to create a deployment for VM

There are plenty of resources on host: cores, RAM, storage.

Pleas help

management-server.log:
2017-10-20 09:10:43,713 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-3:null) (logid:) SeqA 5-377872: Processing Seq
5-377872:  { Cmd , MgmtId: -1, via: 5, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1,"_loadInfo":"{\n
\"connections\": []\n}","wait":0}}] }
2017-10-20 09:10:43,715 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-3:null) (logid:) SeqA 5-377872: Sending Seq
5-377872:  { Ans: , MgmtId: 264216221068220, via: 5, Ver: v1, Flags:
100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2017-10-20 09:10:45,239 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-566e7818) (logid:24eee607) Begin cleanup
expired async-jobs
2017-10-20 09:10:45,243 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-566e7818) (logid:24eee607) End cleanup expired
async-jobs
2017-10-20 09:10:53,566 DEBUG [c.c.a.ApiServlet]
(catalina-exec-6:ctx-3911a4cc) (logid:41db964f) ===START===  192.168.1.194
-- GET
command=deployVirtualMachine=json=b3d5b8bb-2dea-478b-8077-fc813b90d4c6=244d6029-0f9a-4479-b11b-09e4fe6c40e1=KVM=e0ef2da9-a489-4c42-ab6d-c49fd41b0fd2=f39fb02b-4fc0-4f2e-9e4e-dd54f7fc9508=50=1586621a-8e8d-11e7-8f52-f04da2002bbe=mtl-dev04-app02=mtl-dev04-app02=us&_=1508490653550
2017-10-20 09:10:53,581 DEBUG [c.c.u.AccountManagerImpl]
(catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Access granted
to Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin] to
org.apache.cloudstack.quota.vo.ServiceOfferingVO$$EnhancerByCGLIB$$4d79c9fa@540d8fcb
by AffinityGroupAccessChecker
2017-10-20 09:10:53,581 DEBUG [c.c.u.AccountManagerImpl]
(catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Access granted
to Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin] to
com.cloud.storage.DiskOfferingVO$$EnhancerByCGLIB$$f64346ed@7d0dcb96 by
AffinityGroupAccessChecker
2017-10-20 09:10:53,605 DEBUG [c.c.v.UserVmManagerImpl]
(catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Allocating in
the DB for vm
2017-10-20 09:10:53,613 DEBUG [c.c.v.VirtualMachineManagerImpl]
(catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Allocating
entries for VM: VM[User|i-2-23-VM]
2017-10-20 09:10:53,614 DEBUG [c.c.v.VirtualMachineManagerImpl]
(catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Allocating
nics for VM[User|i-2-23-VM]
2017-10-20 09:10:53,614 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Allocating nic
for vm VM[User|i-2-23-VM] in network Ntwk[204|Guest|6] with requested
profile NicProfile[0-0-null-null-null
2017-10-20 09:10:53,619 DEBUG [c.c.v.VirtualMachineManagerImpl]
(catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Allocating
disks for VM[User|i-2-23-VM]
2017-10-20 09:10:53,626 DEBUG [c.c.v.VirtualMachineManagerImpl]
(catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Allocation
completed for VM: VM[User|i-2-23-VM]
2017-10-20 09:10:53,626 DEBUG [c.c.v.UserVmManagerImpl]
(catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) Successfully
allocated DB entry for VM[User|i-2-23-VM]
2017-10-20 09:10:53,653 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-40:ctx-a4cff151 job-181) (logid:0c9b9886) Add job-181
into job monitoring
2017-10-20 09:10:53,658 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f) submit async
job-181, details: AsyncJobVO {id:181, userId: 2, accountId: 2,
instanceType: VirtualMachine, instanceId: 23, cmd:
org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin, cmdInfo:
{"keyboard":"us","httpmethod":"GET","templateid":"244d6029-0f9a-4479-b11b-09e4fe6c40e1","securitygroupids":"1586621a-8e8d-11e7-8f52-f04da2002bbe","ctxAccountId":"2","uuid":"df7da1f1-4f17-4e98-9950-84f30de1fcb2","cmdEventType":"VM.CREATE","diskofferingid":"f39fb02b-4fc0-4f2e-9e4e-dd54f7fc9508","size":"50","serviceofferingid":"e0ef2da9-a489-4c42-ab6d-c49fd41b0fd2","response":"json","ctxUserId":"2","hypervisor":"KVM","displayname":"mtl-dev04-app02","name":"mtl-dev04-app02","zoneid":"b3d5b8bb-2dea-478b-8077-fc813b90d4c6","ctxStartEventId":"277","id":"23","ctxDetails":"{\"interface
com.cloud.offering.DiskOffering\":\"f39fb02b-4fc0-4f2e-9e4e-dd54f7fc9508\",\"interface
com.cloud.network.security.SecurityGroup\":\"1586621a-8e8d-11e7-8f52-f04da2002bbe\",\"interface
com.cloud.vm.VirtualMachine\":\"df7da1f1-4f17-4e98-9950-84f30de1fcb2\",\"interface
com.cloud.dc.DataCenter\":\"b3d5b8bb-2dea-478b-8077-fc813b90d4c6\",\"interface
com.cloud.offering.ServiceOffering\":\"e0ef2da9-a489-4c42-ab6d-c49fd41b0fd2\",\"interface

Re: Failed to find db.properties

[Jevgeni Zolotarjov]

end of story? no one can help me?

On Thu, Aug 24, 2017 at 8:15 PM, Paul Angus <paul.an...@shapeblue.com>
wrote:

> Ah.
>
> We don't support a Debian based management server - the Debian repo works
> for Ubuntu 14.04 definitely (not sure if we have 16.04 sorted yet).
> For CentOS the current latest is 7.3
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: Jevgeni Zolotarjov [mailto:j.zolotar...@gmail.com]
> Sent: 24 August 2017 18:13
> To: users@cloudstack.apache.org
> Subject: Re: Failed to find db.properties
>
> Debian 8.9
> the service is called tomcat7 in debian - not tomcat
>
> On Thu, Aug 24, 2017 at 8:10 PM, Paul Angus <paul.an...@shapeblue.com>
> wrote:
>
> > Hi Jevgeni,
> >
> > What OS are you running your management server on?  (sorry if it's buried
> > in your post somewhere).
> > The log suggest that tomcat7 isn't installed, or isn't where CloudStack
> is
> > expecting it to be.
> >
> >
> > Kind regards,
> >
> > Paul Angus
> >
> > paul.an...@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> > -Original Message-
> > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com]
> > Sent: 24 August 2017 18:06
> > To: users@cloudstack.apache.org
> > Subject: Re: Failed to find db.properties
> >
> > It seems that there is something wrong with your installation. Maybe a
> > corrupted file?
> >
> > On Thu, Aug 24, 2017 at 1:04 PM, Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com>
> > wrote:
> >
> > > sorry. this log file is correct
> > >
> > > DEBUG:root:execute:/usr/bin/lsb_release -i
> > > DEBUG:root:execute:hostname -f
> > > DEBUG:root:execute:iptables-save|grep INPUT|grep -w 8080
> > > DEBUG:root:Failed to execute:
> > > DEBUG:root:execute:ufw allow 8080/tcp
> > > DEBUG:root:execute:iptables-save|grep INPUT|grep -w 8250
> > > DEBUG:root:Failed to execute:
> > > DEBUG:root:execute:ufw allow 8250/tcp
> > > DEBUG:root:execute:iptables-save|grep INPUT|grep -w 9090
> > > DEBUG:root:Failed to execute:
> > > DEBUG:root:execute:ufw allow 9090/tcp
> > > DEBUG:root:execute:rm -f /etc/cloudstack/management/server.xml
> > > DEBUG:root:execute:ln -s /etc/cloudstack/management/server7-nonssl.xml
> > > /etc/cloudstack/management/server.xml
> > > DEBUG:root:execute:rm -f /usr/share/cloudstack-management/bin
> > > DEBUG:root:execute:ln -s /usr/share/tomcat7/bin
> > > /usr/share/cloudstack-management/bin
> > > DEBUG:root:execute:rm -f /usr/share/cloudstack-management/lib
> > > DEBUG:root:execute:ln -s /usr/share/tomcat7/lib
> > > /usr/share/cloudstack-management/lib
> > > DEBUG:root:execute:touch /var/run/cloudstack-management.pid
> > > DEBUG:root:execute:chown cloud.cloud /var/run/cloudstack-
> management.pid
> > > DEBUG:root:execute:hostname --fqdn
> > > DEBUG:root:execute:chown cloud:cloud -R /var/lib/cloudstack/
> > > DEBUG:root:execute:chmod +x -R
> > > /usr/share/cloudstack-management/webapps/client/WEB-
> INF/classes/scripts/
> > > DEBUG:root:execute:sudo /usr/sbin/service tomcat status
> > > DEBUG:root:Failed to execute:● tomcat.service
> > >Loaded: not-found (Reason: No such file or directory)
> > >Active: inactive (dead)
> > > DEBUG:root:execute:sudo /usr/sbin/service tomcat stop
> > > DEBUG:root:Failed to execute:Failed to stop tomcat.service: Unit
> > > tomcat.service not loaded.
> > > DEBUG:root:execute:sudo update-rc.d -f tomcat remove
> > > DEBUG:root:execute:sudo /usr/sbin/service cloudstack-management status
> > > DEBUG:root:Failed to execute:● cloudstack-management.service -
> CloudStack
> > > Management Server
> > >Loaded: loaded (/lib/systemd/system/cloudstack-management.service;
> > > enabled)
> > >Active: failed (Result: exit-code) since Thu 2017-08-24 18:15:22
> CEST;
> > > 15min ago
> > >  Main PID: 19262 (code=exited, status=1/FAILURE)
> > >
> > > Aug 24 18:15:22 servername jsvc.exec[19263]: at
> > > org.apache.catalina.startup.Bootstrap.init(Bootstrap.java:235)
> > > Aug 24 18:15:22 servername jsvc.exec[19263]: at
> > > org.apache.catalina.startup.Bootstrap.init(Bootstrap.java:307)
> > >

Re: Failed to find db.properties

[Jevgeni Zolotarjov]

Debian 8.9
the service is called tomcat7 in debian - not tomcat

On Thu, Aug 24, 2017 at 8:10 PM, Paul Angus <paul.an...@shapeblue.com>
wrote:

> Hi Jevgeni,
>
> What OS are you running your management server on?  (sorry if it's buried
> in your post somewhere).
> The log suggest that tomcat7 isn't installed, or isn't where CloudStack is
> expecting it to be.
>
>
> Kind regards,
>
> Paul Angus
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com]
> Sent: 24 August 2017 18:06
> To: users@cloudstack.apache.org
> Subject: Re: Failed to find db.properties
>
> It seems that there is something wrong with your installation. Maybe a
> corrupted file?
>
> On Thu, Aug 24, 2017 at 1:04 PM, Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> wrote:
>
> > sorry. this log file is correct
> >
> > DEBUG:root:execute:/usr/bin/lsb_release -i
> > DEBUG:root:execute:hostname -f
> > DEBUG:root:execute:iptables-save|grep INPUT|grep -w 8080
> > DEBUG:root:Failed to execute:
> > DEBUG:root:execute:ufw allow 8080/tcp
> > DEBUG:root:execute:iptables-save|grep INPUT|grep -w 8250
> > DEBUG:root:Failed to execute:
> > DEBUG:root:execute:ufw allow 8250/tcp
> > DEBUG:root:execute:iptables-save|grep INPUT|grep -w 9090
> > DEBUG:root:Failed to execute:
> > DEBUG:root:execute:ufw allow 9090/tcp
> > DEBUG:root:execute:rm -f /etc/cloudstack/management/server.xml
> > DEBUG:root:execute:ln -s /etc/cloudstack/management/server7-nonssl.xml
> > /etc/cloudstack/management/server.xml
> > DEBUG:root:execute:rm -f /usr/share/cloudstack-management/bin
> > DEBUG:root:execute:ln -s /usr/share/tomcat7/bin
> > /usr/share/cloudstack-management/bin
> > DEBUG:root:execute:rm -f /usr/share/cloudstack-management/lib
> > DEBUG:root:execute:ln -s /usr/share/tomcat7/lib
> > /usr/share/cloudstack-management/lib
> > DEBUG:root:execute:touch /var/run/cloudstack-management.pid
> > DEBUG:root:execute:chown cloud.cloud /var/run/cloudstack-management.pid
> > DEBUG:root:execute:hostname --fqdn
> > DEBUG:root:execute:chown cloud:cloud -R /var/lib/cloudstack/
> > DEBUG:root:execute:chmod +x -R
> > /usr/share/cloudstack-management/webapps/client/WEB-INF/classes/scripts/
> > DEBUG:root:execute:sudo /usr/sbin/service tomcat status
> > DEBUG:root:Failed to execute:● tomcat.service
> >Loaded: not-found (Reason: No such file or directory)
> >Active: inactive (dead)
> > DEBUG:root:execute:sudo /usr/sbin/service tomcat stop
> > DEBUG:root:Failed to execute:Failed to stop tomcat.service: Unit
> > tomcat.service not loaded.
> > DEBUG:root:execute:sudo update-rc.d -f tomcat remove
> > DEBUG:root:execute:sudo /usr/sbin/service cloudstack-management status
> > DEBUG:root:Failed to execute:● cloudstack-management.service - CloudStack
> > Management Server
> >Loaded: loaded (/lib/systemd/system/cloudstack-management.service;
> > enabled)
> >Active: failed (Result: exit-code) since Thu 2017-08-24 18:15:22 CEST;
> > 15min ago
> >  Main PID: 19262 (code=exited, status=1/FAILURE)
> >
> > Aug 24 18:15:22 servername jsvc.exec[19263]: at
> > org.apache.catalina.startup.Bootstrap.init(Bootstrap.java:235)
> > Aug 24 18:15:22 servername jsvc.exec[19263]: at
> > org.apache.catalina.startup.Bootstrap.init(Bootstrap.java:307)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > Aug 24 18:15:22 servername jsvc.exec[19263]: at
> > sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:
> > 62)
> > Aug 24 18:15:22 servername jsvc.exec[19263]: at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:43)
> > Aug 24 18:15:22 servername jsvc.exec[19263]: at
> > java.lang.reflect.Method.invoke(Method.java:498)
> > Aug 24 18:15:22 servername jsvc.exec[19263]: at
> > org.apache.commons.daemon.support.DaemonLoader.load(
> DaemonLoader.java:212)
> > Aug 24 18:15:22 servername jsvc.exec[19263]: 2017-08-24 18:15:22 19264
> > jsvc.exec error: Cannot load daemon
> > Aug 24 18:15:22 servername systemd[1]: cloudstack-management.service:
> main
> > process exited, code=exited, status=1/FAILURE
> > Aug 24 18:15:22 servername systemd[1]: cloudstack-management.service:
> > control process exited, code=exited status=255
> > Aug 24 18:15:22 servername systemd[1]: Unit cloudstack-management.service

Re: Failed to find db.properties

[Jevgeni Zolotarjov]

how something can be wrong?
I am installing to virgin clean Debian 8.9 machine
using these repository
http://packages.shapeblue.com/cloudstack/upstream/debian/4.10/
and the guide found on cloudstack website.

The only thing, I did - was installation of Java8

On Thu, Aug 24, 2017 at 8:05 PM, Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

> It seems that there is something wrong with your installation. Maybe a
> corrupted file?
>
> On Thu, Aug 24, 2017 at 1:04 PM, Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> wrote:
>
> > sorry. this log file is correct
> >
> > DEBUG:root:execute:/usr/bin/lsb_release -i
> > DEBUG:root:execute:hostname -f
> > DEBUG:root:execute:iptables-save|grep INPUT|grep -w 8080
> > DEBUG:root:Failed to execute:
> > DEBUG:root:execute:ufw allow 8080/tcp
> > DEBUG:root:execute:iptables-save|grep INPUT|grep -w 8250
> > DEBUG:root:Failed to execute:
> > DEBUG:root:execute:ufw allow 8250/tcp
> > DEBUG:root:execute:iptables-save|grep INPUT|grep -w 9090
> > DEBUG:root:Failed to execute:
> > DEBUG:root:execute:ufw allow 9090/tcp
> > DEBUG:root:execute:rm -f /etc/cloudstack/management/server.xml
> > DEBUG:root:execute:ln -s /etc/cloudstack/management/server7-nonssl.xml
> > /etc/cloudstack/management/server.xml
> > DEBUG:root:execute:rm -f /usr/share/cloudstack-management/bin
> > DEBUG:root:execute:ln -s /usr/share/tomcat7/bin
> > /usr/share/cloudstack-management/bin
> > DEBUG:root:execute:rm -f /usr/share/cloudstack-management/lib
> > DEBUG:root:execute:ln -s /usr/share/tomcat7/lib
> > /usr/share/cloudstack-management/lib
> > DEBUG:root:execute:touch /var/run/cloudstack-management.pid
> > DEBUG:root:execute:chown cloud.cloud /var/run/cloudstack-management.pid
> > DEBUG:root:execute:hostname --fqdn
> > DEBUG:root:execute:chown cloud:cloud -R /var/lib/cloudstack/
> > DEBUG:root:execute:chmod +x -R
> > /usr/share/cloudstack-management/webapps/client/WEB-INF/classes/scripts/
> > DEBUG:root:execute:sudo /usr/sbin/service tomcat status
> > DEBUG:root:Failed to execute:● tomcat.service
> >Loaded: not-found (Reason: No such file or directory)
> >Active: inactive (dead)
> > DEBUG:root:execute:sudo /usr/sbin/service tomcat stop
> > DEBUG:root:Failed to execute:Failed to stop tomcat.service: Unit
> > tomcat.service not loaded.
> > DEBUG:root:execute:sudo update-rc.d -f tomcat remove
> > DEBUG:root:execute:sudo /usr/sbin/service cloudstack-management status
> > DEBUG:root:Failed to execute:● cloudstack-management.service - CloudStack
> > Management Server
> >Loaded: loaded (/lib/systemd/system/cloudstack-management.service;
> > enabled)
> >Active: failed (Result: exit-code) since Thu 2017-08-24 18:15:22 CEST;
> > 15min ago
> >  Main PID: 19262 (code=exited, status=1/FAILURE)
> >
> > Aug 24 18:15:22 servername jsvc.exec[19263]: at
> > org.apache.catalina.startup.Bootstrap.init(Bootstrap.java:235)
> > Aug 24 18:15:22 servername jsvc.exec[19263]: at
> > org.apache.catalina.startup.Bootstrap.init(Bootstrap.java:307)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > Aug 24 18:15:22 servername jsvc.exec[19263]: at
> > sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:
> > 62)
> > Aug 24 18:15:22 servername jsvc.exec[19263]: at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:43)
> > Aug 24 18:15:22 servername jsvc.exec[19263]: at
> > java.lang.reflect.Method.invoke(Method.java:498)
> > Aug 24 18:15:22 servername jsvc.exec[19263]: at
> > org.apache.commons.daemon.support.DaemonLoader.load(
> DaemonLoader.java:212)
> > Aug 24 18:15:22 servername jsvc.exec[19263]: 2017-08-24 18:15:22 19264
> > jsvc.exec error: Cannot load daemon
> > Aug 24 18:15:22 servername systemd[1]: cloudstack-management.service:
> main
> > process exited, code=exited, status=1/FAILURE
> > Aug 24 18:15:22 servername systemd[1]: cloudstack-management.service:
> > control process exited, code=exited status=255
> > Aug 24 18:15:22 servername systemd[1]: Unit cloudstack-management.service
> > entered failed state.
> > DEBUG:root:execute:sudo /usr/sbin/service cloudstack-management stop
> > DEBUG:root:execute:sudo update-rc.d -f cloudstack-management remove
> > DEBUG:root:execute:sudo update-rc.d -f cloudstack-management defaults
> > DEBUG:root:execute:sudo /usr/sbin/service cloudstack-management status
> > DEBUG:root:Failed to execute:● cloudstack-management.service - CloudStack
> > Manage

Re: Failed to find db.properties

[Jevgeni Zolotarjov]

-management.service
entered failed state.
Aug 24 18:31:14 servername systemd[1]: Stopped CloudStack Management Server.
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-management start


On Thu, Aug 24, 2017 at 8:02 PM, Jevgeni Zolotarjov <j.zolotar...@gmail.com>
wrote:

> Aug 22, 2017 7:34:17 PM org.apache.catalina.startup.ClassLoaderFactory
> validateFile
> WARNING: Problem with directory [/usr/share/tomcat7/common/classes],
> exists: [false], isDirectory: [false], canRead: [false]
> Aug 22, 2017 7:34:17 PM org.apache.catalina.startup.ClassLoaderFactory
> validateFile
> WARNING: Problem with directory [/usr/share/tomcat7/common], exists:
> [false], isDirectory: [false], canRead: [false]
> Aug 22, 2017 7:34:17 PM org.apache.catalina.startup.ClassLoaderFactory
> validateFile
> WARNING: Problem with directory [/usr/share/tomcat7/server/classes],
> exists: [false], isDirectory: [false], canRead: [false]
> Aug 22, 2017 7:34:17 PM org.apache.catalina.startup.ClassLoaderFactory
> validateFile
> WARNING: Problem with directory [/usr/share/tomcat7/server], exists:
> [false], isDirectory: [false], canRead: [false]
> Aug 22, 2017 7:34:17 PM org.apache.catalina.startup.ClassLoaderFactory
> validateFile
> WARNING: Problem with directory [/usr/share/tomcat7/shared/classes],
> exists: [false], isDirectory: [false], canRead: [false]
> Aug 22, 2017 7:34:17 PM org.apache.catalina.startup.ClassLoaderFactory
> validateFile
> WARNING: Problem with directory [/usr/share/tomcat7/shared], exists:
> [false], isDirectory: [false], canRead: [false]
> Aug 22, 2017 7:34:18 PM org.apache.coyote.AbstractProtocol init
> INFO: Initializing ProtocolHandler ["http-bio-8080"]
> Aug 22, 2017 7:34:18 PM org.apache.catalina.startup.Catalina load
> INFO: Initialization processed in 345 ms
> Aug 22, 2017 7:34:18 PM org.apache.catalina.core.StandardService
> startInternal
> INFO: Starting service Catalina
> Aug 22, 2017 7:34:18 PM org.apache.catalina.core.StandardEngine
> startInternal
> INFO: Starting Servlet Engine: Apache Tomcat/7.0.56 (Debian)
> Aug 22, 2017 7:34:18 PM org.apache.catalina.startup.HostConfig
> deployDirectory
> INFO: Deploying web application directory /var/lib/tomcat7/webapps/ROOT
> Aug 22, 2017 7:34:18 PM org.apache.catalina.startup.HostConfig
> deployDirectory
> INFO: Deployment of web application directory
> /var/lib/tomcat7/webapps/ROOT has finished in 398 ms
> Aug 22, 2017 7:34:18 PM org.apache.coyote.AbstractProtocol start
> INFO: Starting ProtocolHandler ["http-bio-8080"]
> Aug 22, 2017 7:34:18 PM org.apache.catalina.startup.Catalina start
> INFO: Server startup in 437 ms
> Aug 22, 2017 7:34:23 PM org.apache.coyote.AbstractProtocol pause
> INFO: Pausing ProtocolHandler ["http-bio-8080"]
> Aug 22, 2017 7:34:23 PM org.apache.catalina.core.StandardService
> stopInternal
> INFO: Stopping service Catalina
> Aug 22, 2017 7:34:23 PM org.apache.coyote.AbstractProtocol stop
> INFO: Stopping ProtocolHandler ["http-bio-8080"]
> Aug 22, 2017 7:34:23 PM org.apache.coyote.AbstractProtocol destroy
> INFO: Destroying ProtocolHandler ["http-bio-8080"]
> Aug 22, 2017 7:42:24 PM org.apache.catalina.startup.ClassLoaderFactory
> validateFile
> WARNING: Problem with directory [/usr/share/tomcat7/common/classes],
> exists: [false], isDirectory: [false], canRead: [false]
> Aug 22, 2017 7:42:24 PM org.apache.catalina.startup.ClassLoaderFactory
> validateFile
> WARNING: Problem with directory [/usr/share/tomcat7/common], exists:
> [false], isDirectory: [false], canRead: [false]
> Aug 22, 2017 7:42:25 PM org.apache.catalina.startup.ClassLoaderFactory
> validateFile
> WARNING: Problem with directory [/usr/share/tomcat7/server/classes],
> exists: [false], isDirectory: [false], canRead: [false]
> Aug 22, 2017 7:42:25 PM org.apache.catalina.startup.ClassLoaderFactory
> validateFile
> WARNING: Problem with directory [/usr/share/tomcat7/server], exists:
> [false], isDirectory: [false], canRead: [false]
> Aug 22, 2017 7:42:25 PM org.apache.catalina.startup.ClassLoaderFactory
> validateFile
> WARNING: Problem with directory [/usr/share/tomcat7/shared/classes],
> exists: [false], isDirectory: [false], canRead: [false]
> Aug 22, 2017 7:42:25 PM org.apache.catalina.startup.ClassLoaderFactory
> validateFile
> WARNING: Problem with directory [/usr/share/tomcat7/shared], exists:
> [false], isDirectory: [false], canRead: [false]
> Aug 22, 2017 7:42:26 PM org.apache.coyote.AbstractProtocol init
> INFO: Initializing ProtocolHandler ["http-bio-8080"]
> Aug 22, 2017 7:42:26 PM org.apache.catalina.startup.Catalina load
> INFO: Initialization processed in 1158 ms
> Aug 22, 2017 7:42:26 PM org.apache.catalina.core.StandardService
> startInternal
>

Re: Failed to find db.properties

[Jevgeni Zolotarjov]

Aug 22, 2017 7:34:17 PM org.apache.catalina.startup.ClassLoaderFactory
validateFile
WARNING: Problem with directory [/usr/share/tomcat7/common/classes],
exists: [false], isDirectory: [false], canRead: [false]
Aug 22, 2017 7:34:17 PM org.apache.catalina.startup.ClassLoaderFactory
validateFile
WARNING: Problem with directory [/usr/share/tomcat7/common], exists:
[false], isDirectory: [false], canRead: [false]
Aug 22, 2017 7:34:17 PM org.apache.catalina.startup.ClassLoaderFactory
validateFile
WARNING: Problem with directory [/usr/share/tomcat7/server/classes],
exists: [false], isDirectory: [false], canRead: [false]
Aug 22, 2017 7:34:17 PM org.apache.catalina.startup.ClassLoaderFactory
validateFile
WARNING: Problem with directory [/usr/share/tomcat7/server], exists:
[false], isDirectory: [false], canRead: [false]
Aug 22, 2017 7:34:17 PM org.apache.catalina.startup.ClassLoaderFactory
validateFile
WARNING: Problem with directory [/usr/share/tomcat7/shared/classes],
exists: [false], isDirectory: [false], canRead: [false]
Aug 22, 2017 7:34:17 PM org.apache.catalina.startup.ClassLoaderFactory
validateFile
WARNING: Problem with directory [/usr/share/tomcat7/shared], exists:
[false], isDirectory: [false], canRead: [false]
Aug 22, 2017 7:34:18 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8080"]
Aug 22, 2017 7:34:18 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 345 ms
Aug 22, 2017 7:34:18 PM org.apache.catalina.core.StandardService
startInternal
INFO: Starting service Catalina
Aug 22, 2017 7:34:18 PM org.apache.catalina.core.StandardEngine
startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.56 (Debian)
Aug 22, 2017 7:34:18 PM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory /var/lib/tomcat7/webapps/ROOT
Aug 22, 2017 7:34:18 PM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deployment of web application directory /var/lib/tomcat7/webapps/ROOT
has finished in 398 ms
Aug 22, 2017 7:34:18 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8080"]
Aug 22, 2017 7:34:18 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 437 ms
Aug 22, 2017 7:34:23 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-8080"]
Aug 22, 2017 7:34:23 PM org.apache.catalina.core.StandardService
stopInternal
INFO: Stopping service Catalina
Aug 22, 2017 7:34:23 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-8080"]
Aug 22, 2017 7:34:23 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-8080"]
Aug 22, 2017 7:42:24 PM org.apache.catalina.startup.ClassLoaderFactory
validateFile
WARNING: Problem with directory [/usr/share/tomcat7/common/classes],
exists: [false], isDirectory: [false], canRead: [false]
Aug 22, 2017 7:42:24 PM org.apache.catalina.startup.ClassLoaderFactory
validateFile
WARNING: Problem with directory [/usr/share/tomcat7/common], exists:
[false], isDirectory: [false], canRead: [false]
Aug 22, 2017 7:42:25 PM org.apache.catalina.startup.ClassLoaderFactory
validateFile
WARNING: Problem with directory [/usr/share/tomcat7/server/classes],
exists: [false], isDirectory: [false], canRead: [false]
Aug 22, 2017 7:42:25 PM org.apache.catalina.startup.ClassLoaderFactory
validateFile
WARNING: Problem with directory [/usr/share/tomcat7/server], exists:
[false], isDirectory: [false], canRead: [false]
Aug 22, 2017 7:42:25 PM org.apache.catalina.startup.ClassLoaderFactory
validateFile
WARNING: Problem with directory [/usr/share/tomcat7/shared/classes],
exists: [false], isDirectory: [false], canRead: [false]
Aug 22, 2017 7:42:25 PM org.apache.catalina.startup.ClassLoaderFactory
validateFile
WARNING: Problem with directory [/usr/share/tomcat7/shared], exists:
[false], isDirectory: [false], canRead: [false]
Aug 22, 2017 7:42:26 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8080"]
Aug 22, 2017 7:42:26 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1158 ms
Aug 22, 2017 7:42:26 PM org.apache.catalina.core.StandardService
startInternal
INFO: Starting service Catalina
Aug 22, 2017 7:42:26 PM org.apache.catalina.core.StandardEngine
startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.56 (Debian)
Aug 22, 2017 7:42:27 PM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory /var/lib/tomcat7/webapps/ROOT
Aug 22, 2017 7:42:29 PM org.apache.catalina.util.SessionIdGenerator
createSecureRandom
INFO: Creation of SecureRandom instance for session ID generation using
[SHA1PRNG] took [1,376] milliseconds.
Aug 22, 2017 7:42:29 PM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deployment of web application directory /var/lib/tomcat7/webapps/ROOT
has finished in 1,902 ms
Aug 22, 2017 7:42:29 PM org.apache.coyote.AbstractProtocol start

Re: Failed to find db.properties

[Jevgeni Zolotarjov]

I guess, this management setup log file explains the root of evil.
There are many errors, though setup reports everything is OK.



On Thu, Aug 24, 2017 at 4:12 PM, Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

> db.properties is there.
> Can you post your log files somewhere?
>
> On Thu, Aug 24, 2017 at 8:51 AM, Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> wrote:
>
> > This is what I have in
> > /etc/cloudstack/
> > /etc/cloudstack/management/
> >
> > root@servername:/var/cache/tomcat7/Catalina/localhost/client# cd
> > /etc/cloudstack/
> > root@servername:/etc/cloudstack# ls -al
> > total 20
> > drwxr-xr-x   5 root root 4096 Aug 23 16:31 .
> > drwxr-xr-x 100 root root 4096 Aug 23 19:26 ..
> > drwxr-xr-x   2 root root 4096 Aug 23 14:27 agent
> > drwxr-xr-x   2 root root 4096 Aug 24 09:23 management
> > drwxr-xr-x   2 root root 4096 Aug 22 19:34 usage
> > root@servername:/etc/cloudstack# cd management/
> > root@servername:/etc/cloudstack/management# ls -al
> > total 176
> > drwxr-xr-x 2 root root   4096 Aug 24 09:23 .
> > drwxr-xr-x 5 root root   4096 Aug 23 16:31 ..
> > -rw-r--r-- 1 root root   8945 Jul 12 12:33 catalina.policy
> > -rw-r--r-- 1 root root   3795 Jul 12 12:33 catalina.properties
> > -rw-r--r-- 1 root root   1649 Jul 12 12:33 classpath.conf
> > -rw-r--r-- 1 root root   1435 Jul 12 12:33 context.xml
> > -rw-r- 1 root cloud  3273 Jul 12 12:33 db.properties
> > -rw-r--r-- 1 root root  23292 Jul 12 12:33 ehcache.xml
> > -rw-r--r-- 1 root root976 Jul 12 12:33 environment.properties
> > -rw-r--r-- 1 root root927 Jul 12 12:33 java.security.ciphers
> > -rw-r--r-- 1 root root   7467 Jul 12 12:33 log4j-cloud.xml
> > -rw-r--r-- 1 root root   3257 Jul 12 12:33 logging.properties
> > -rw-r--r-- 1 root root   6722 Jul 12 12:33 server7-nonssl.xml
> > -rw-r--r-- 1 root root   7251 Jul 12 12:33 server7-ssl.xml
> > -rw-r--r-- 1 root root   6767 Jul 12 12:33 server-nonssl.xml
> > -rw-r--r-- 1 root root   7300 Jul 12 12:33 server-ssl.xml
> > lrwxrwxrwx 1 root root 45 Aug 24 09:23 server.xml ->
> > /etc/cloudstack/management/server7-nonssl.xml
> > lrwxrwxrwx 1 root root 46 Aug 23 16:37 tomcat6.conf ->
> > /etc/cloudstack/management/tomcat6-nonssl.conf
> > -rw-r--r-- 1 root root   2692 Jul 12 12:33 tomcat6-nonssl.conf
> > -rw-r--r-- 1 root root   2821 Jul 12 12:33 tomcat6-ssl.conf
> > -rw-r--r-- 1 root root   1383 Jul 12 12:33 tomcat-users.xml
> > -rw-r--r-- 1 root root  50428 Jul 12 12:33 web.xml
> >
> >
> > On Thu, Aug 24, 2017 at 2:13 PM, Rafael Weingärtner <
> > rafaelweingart...@gmail.com> wrote:
> >
> > > Is there a db.properties in /etc/cloudstack.. folder?
> > >
> > > On Thu, Aug 24, 2017 at 3:27 AM, Jevgeni Zolotarjov <
> > > j.zolotar...@gmail.com>
> > > wrote:
> > >
> > > > Did that "setup --tomcat7" now.
> > > > Output:
> > > > Starting to configure CloudStack Management Server:
> > > > Configure Firewall ...[OK]
> > > > Configure CloudStack Management Server ...[OK]
> > > > CloudStack Management Server setup is Done!
> > > >
> > > > But no difference, when I try to run client application. Absolutely
> the
> > > > same result in catalina.log - Failed to find db.properties
> > > >
> > > >
> > > >
> > > > On Thu, Aug 24, 2017 at 1:24 AM, ilya <ilya.mailing.li...@gmail.com>
> > > > wrote:
> > > >
> > > > > Rafael has a point there - please try with tomcat7
> > > > >
> > > > > Also - please look into catalina.out/log for further debugging
> info.
> > > > >
> > > > > On 8/23/17 9:23 AM, Rafael Weingärtner wrote:
> > > > > > did you run setup-management --tomcat7?
> > > > > >
> > > > > > On Wed, Aug 23, 2017 at 12:20 PM, Jevgeni Zolotarjov <
> > > > > j.zolotar...@gmail.com
> > > > > >> wrote:
> > > > > >
> > > > > >> Yes, I did, as it is suggested in the guide:
> > > > > >> cloudstack-setup-database then setup-management
> > > > > >>
> > > > > >>
> > > > > >> On Aug 23, 2017 7:11 PM, "Rafael Weingärtner" <
> > > > > rafaelweingart...@gmail.com
> > > > > >>>
> > > > > >> wrote:
> > > > > >>

Re: Failed to find db.properties

[Jevgeni Zolotarjov]

This is what I have in
/etc/cloudstack/
/etc/cloudstack/management/

root@servername:/var/cache/tomcat7/Catalina/localhost/client# cd
/etc/cloudstack/
root@servername:/etc/cloudstack# ls -al
total 20
drwxr-xr-x   5 root root 4096 Aug 23 16:31 .
drwxr-xr-x 100 root root 4096 Aug 23 19:26 ..
drwxr-xr-x   2 root root 4096 Aug 23 14:27 agent
drwxr-xr-x   2 root root 4096 Aug 24 09:23 management
drwxr-xr-x   2 root root 4096 Aug 22 19:34 usage
root@servername:/etc/cloudstack# cd management/
root@servername:/etc/cloudstack/management# ls -al
total 176
drwxr-xr-x 2 root root   4096 Aug 24 09:23 .
drwxr-xr-x 5 root root   4096 Aug 23 16:31 ..
-rw-r--r-- 1 root root   8945 Jul 12 12:33 catalina.policy
-rw-r--r-- 1 root root   3795 Jul 12 12:33 catalina.properties
-rw-r--r-- 1 root root   1649 Jul 12 12:33 classpath.conf
-rw-r--r-- 1 root root   1435 Jul 12 12:33 context.xml
-rw-r- 1 root cloud  3273 Jul 12 12:33 db.properties
-rw-r--r-- 1 root root  23292 Jul 12 12:33 ehcache.xml
-rw-r--r-- 1 root root976 Jul 12 12:33 environment.properties
-rw-r--r-- 1 root root927 Jul 12 12:33 java.security.ciphers
-rw-r--r-- 1 root root   7467 Jul 12 12:33 log4j-cloud.xml
-rw-r--r-- 1 root root   3257 Jul 12 12:33 logging.properties
-rw-r--r-- 1 root root   6722 Jul 12 12:33 server7-nonssl.xml
-rw-r--r-- 1 root root   7251 Jul 12 12:33 server7-ssl.xml
-rw-r--r-- 1 root root   6767 Jul 12 12:33 server-nonssl.xml
-rw-r--r-- 1 root root   7300 Jul 12 12:33 server-ssl.xml
lrwxrwxrwx 1 root root 45 Aug 24 09:23 server.xml ->
/etc/cloudstack/management/server7-nonssl.xml
lrwxrwxrwx 1 root root 46 Aug 23 16:37 tomcat6.conf ->
/etc/cloudstack/management/tomcat6-nonssl.conf
-rw-r--r-- 1 root root   2692 Jul 12 12:33 tomcat6-nonssl.conf
-rw-r--r-- 1 root root   2821 Jul 12 12:33 tomcat6-ssl.conf
-rw-r--r-- 1 root root   1383 Jul 12 12:33 tomcat-users.xml
-rw-r--r-- 1 root root  50428 Jul 12 12:33 web.xml


On Thu, Aug 24, 2017 at 2:13 PM, Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

> Is there a db.properties in /etc/cloudstack.. folder?
>
> On Thu, Aug 24, 2017 at 3:27 AM, Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> wrote:
>
> > Did that "setup --tomcat7" now.
> > Output:
> > Starting to configure CloudStack Management Server:
> > Configure Firewall ...[OK]
> > Configure CloudStack Management Server ...[OK]
> > CloudStack Management Server setup is Done!
> >
> > But no difference, when I try to run client application. Absolutely the
> > same result in catalina.log - Failed to find db.properties
> >
> >
> >
> > On Thu, Aug 24, 2017 at 1:24 AM, ilya <ilya.mailing.li...@gmail.com>
> > wrote:
> >
> > > Rafael has a point there - please try with tomcat7
> > >
> > > Also - please look into catalina.out/log for further debugging info.
> > >
> > > On 8/23/17 9:23 AM, Rafael Weingärtner wrote:
> > > > did you run setup-management --tomcat7?
> > > >
> > > > On Wed, Aug 23, 2017 at 12:20 PM, Jevgeni Zolotarjov <
> > > j.zolotar...@gmail.com
> > > >> wrote:
> > > >
> > > >> Yes, I did, as it is suggested in the guide:
> > > >> cloudstack-setup-database then setup-management
> > > >>
> > > >>
> > > >> On Aug 23, 2017 7:11 PM, "Rafael Weingärtner" <
> > > rafaelweingart...@gmail.com
> > > >>>
> > > >> wrote:
> > > >>
> > > >> did you run cloudstack-setup-database and ...setup-management?
> > > >>
> > > >>
> > > >> On Wed, Aug 23, 2017 at 11:16 AM, Jevgeni Zolotarjov <
> > > >> j.zolotar...@gmail.com
> > > >>> wrote:
> > > >>
> > > >>> Here you go
> > > >>>
> > > >>> management-server.log is not available, which probably means, the
> > > >>> application did not reach that far and exited due to missing DB
> > access
> > > >>>
> > > >>> On Wed, Aug 23, 2017 at 6:03 PM, Gabriel Beims Bräscher <
> > > >>> gabrasc...@gmail.com> wrote:
> > > >>>
> > > >>>> Hi Jevgeni,
> > > >>>>
> > > >>>> Can you share with us some log files (catalina.log,
> > > >> management-server.log
> > > >>>> and others that might be useful)?
> > > >>>>
> > > >>>>
> > > >>>> 2017-08-23 11:53 GMT-03:00 Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com
> > > >:
> > > >>>>
> > > >>>>> Hi
> > > >>>>>
> > > >>>>> I am installing Cloudstack 4.10 on Debian Jessie using guides
> from
> > > >>>>> http://cloudstack-installation.readthedocs.io/en/latest/
> > > >>>> management-server/
> > > >>>>> index.html
> > > >>>>>
> > > >>>>> Apparently the installation process was smooth. But I cannot get
> > > >> client
> > > >>>>> running on tomcat7
> > > >>>>> From catalina.out log file I can read
> > > >>>>>
> > > >>>>> Failed to find db.properties
> > > >>>>>
> > > >>>>> But this file is present in
> > > >>>>> ./etc/cloudstack/management/db.properties
> > > >>>>> ./etc/cloudstack/usage/db.properties
> > > >>>>>
> > > >>>>> Please help.
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> Rafael Weingärtner
> > > >>
> > > >
> > > >
> > > >
> > >
> >
>
>
>
> --
> Rafael Weingärtner
>

Re: Failed to find db.properties

[Jevgeni Zolotarjov]

Did that "setup --tomcat7" now.
Output:
Starting to configure CloudStack Management Server:
Configure Firewall ...[OK]
Configure CloudStack Management Server ...[OK]
CloudStack Management Server setup is Done!

But no difference, when I try to run client application. Absolutely the
same result in catalina.log - Failed to find db.properties



On Thu, Aug 24, 2017 at 1:24 AM, ilya <ilya.mailing.li...@gmail.com> wrote:

> Rafael has a point there - please try with tomcat7
>
> Also - please look into catalina.out/log for further debugging info.
>
> On 8/23/17 9:23 AM, Rafael Weingärtner wrote:
> > did you run setup-management --tomcat7?
> >
> > On Wed, Aug 23, 2017 at 12:20 PM, Jevgeni Zolotarjov <
> j.zolotar...@gmail.com
> >> wrote:
> >
> >> Yes, I did, as it is suggested in the guide:
> >> cloudstack-setup-database then setup-management
> >>
> >>
> >> On Aug 23, 2017 7:11 PM, "Rafael Weingärtner" <
> rafaelweingart...@gmail.com
> >>>
> >> wrote:
> >>
> >> did you run cloudstack-setup-database and ...setup-management?
> >>
> >>
> >> On Wed, Aug 23, 2017 at 11:16 AM, Jevgeni Zolotarjov <
> >> j.zolotar...@gmail.com
> >>> wrote:
> >>
> >>> Here you go
> >>>
> >>> management-server.log is not available, which probably means, the
> >>> application did not reach that far and exited due to missing DB access
> >>>
> >>> On Wed, Aug 23, 2017 at 6:03 PM, Gabriel Beims Bräscher <
> >>> gabrasc...@gmail.com> wrote:
> >>>
> >>>> Hi Jevgeni,
> >>>>
> >>>> Can you share with us some log files (catalina.log,
> >> management-server.log
> >>>> and others that might be useful)?
> >>>>
> >>>>
> >>>> 2017-08-23 11:53 GMT-03:00 Jevgeni Zolotarjov <j.zolotar...@gmail.com
> >:
> >>>>
> >>>>> Hi
> >>>>>
> >>>>> I am installing Cloudstack 4.10 on Debian Jessie using guides from
> >>>>> http://cloudstack-installation.readthedocs.io/en/latest/
> >>>> management-server/
> >>>>> index.html
> >>>>>
> >>>>> Apparently the installation process was smooth. But I cannot get
> >> client
> >>>>> running on tomcat7
> >>>>> From catalina.out log file I can read
> >>>>>
> >>>>> Failed to find db.properties
> >>>>>
> >>>>> But this file is present in
> >>>>> ./etc/cloudstack/management/db.properties
> >>>>> ./etc/cloudstack/usage/db.properties
> >>>>>
> >>>>> Please help.
> >>>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >> --
> >> Rafael Weingärtner
> >>
> >
> >
> >
>

Re: Failed to find db.properties

[Jevgeni Zolotarjov]

Yes, I did, as it is suggested in the guide:
cloudstack-setup-database then setup-management


On Aug 23, 2017 7:11 PM, "Rafael Weingärtner" <rafaelweingart...@gmail.com>
wrote:

did you run cloudstack-setup-database and ...setup-management?


On Wed, Aug 23, 2017 at 11:16 AM, Jevgeni Zolotarjov <j.zolotar...@gmail.com
> wrote:

> Here you go
>
> management-server.log is not available, which probably means, the
> application did not reach that far and exited due to missing DB access
>
> On Wed, Aug 23, 2017 at 6:03 PM, Gabriel Beims Bräscher <
> gabrasc...@gmail.com> wrote:
>
>> Hi Jevgeni,
>>
>> Can you share with us some log files (catalina.log, management-server.log
>> and others that might be useful)?
>>
>>
>> 2017-08-23 11:53 GMT-03:00 Jevgeni Zolotarjov <j.zolotar...@gmail.com>:
>>
>> > Hi
>> >
>> > I am installing Cloudstack 4.10 on Debian Jessie using guides from
>> > http://cloudstack-installation.readthedocs.io/en/latest/
>> management-server/
>> > index.html
>> >
>> > Apparently the installation process was smooth. But I cannot get client
>> > running on tomcat7
>> > From catalina.out log file I can read
>> >
>> > Failed to find db.properties
>> >
>> > But this file is present in
>> > ./etc/cloudstack/management/db.properties
>> > ./etc/cloudstack/usage/db.properties
>> >
>> > Please help.
>> >
>>
>
>


--
Rafael Weingärtner

Re: Failed to find db.properties

[Jevgeni Zolotarjov]

Here you go

management-server.log is not available, which probably means, the
application did not reach that far and exited due to missing DB access

On Wed, Aug 23, 2017 at 6:03 PM, Gabriel Beims Bräscher <
gabrasc...@gmail.com> wrote:

> Hi Jevgeni,
>
> Can you share with us some log files (catalina.log, management-server.log
> and others that might be useful)?
>
>
> 2017-08-23 11:53 GMT-03:00 Jevgeni Zolotarjov <j.zolotar...@gmail.com>:
>
> > Hi
> >
> > I am installing Cloudstack 4.10 on Debian Jessie using guides from
> > http://cloudstack-installation.readthedocs.io/
> en/latest/management-server/
> > index.html
> >
> > Apparently the installation process was smooth. But I cannot get client
> > running on tomcat7
> > From catalina.out log file I can read
> >
> > Failed to find db.properties
> >
> > But this file is present in
> > ./etc/cloudstack/management/db.properties
> > ./etc/cloudstack/usage/db.properties
> >
> > Please help.
> >
>

Failed to find db.properties

[Jevgeni Zolotarjov]

Hi

I am installing Cloudstack 4.10 on Debian Jessie using guides from
http://cloudstack-installation.readthedocs.io/en/latest/management-server/index.html

Apparently the installation process was smooth. But I cannot get client
running on tomcat7
>From catalina.out log file I can read

Failed to find db.properties

But this file is present in
./etc/cloudstack/management/db.properties
./etc/cloudstack/usage/db.properties

Please help.