Hi All, I've got a CloudStack 4.11.2 lab running on a single host & made the (dumb) mistake of running OS updates without updating the system VMs beforehand. The CloudStack packages upgraded to 4.11.3 just fine but now CloudStack management services wont start (see below).
Here's my question: - Is it *at all* possible to rectify this (now, post package updates) by getting the new system templates in via the CLI tools (/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt)? - The error on the logs *seems* simple, if management could find the new template maybe things could proceed? My impression is no given research I've done so far which seems to amount to: - roll back to 4.11.2 - use the pre-upgrade dump of the DB (which I made) - start management services - get the new system VMs properly (like I should have in the first place) - upgrade to 4.11.3 Humbly (& slightly embarrassed) yours, and thanks, David David Merrill Senior Systems Engineer, Managed and Private/Hybrid Cloud Services OTELCO 92 Oak Street, Portland ME 04101 office 207.772.5678 <callto:207.772.5678> http://www.otelco.com/cloud-and-managed-services Confidentiality Message The information contained in this e-mail transmission may be confidential and legally privileged. If you are not the intended recipient, you are notified that any dissemination, distribution, copying or other use of this information, including attachments, is prohibited. If you received this message in error, please call me at 207.772.5678 <callto:207.772.5678> so this error can be corrected. On 5/17/20, 8:05 AM, "engineer...@reliablenetworks.com" <engineer...@reliablenetworks.com> wrote: 2020-05-17 07:35:15,659 INFO [o.e.j.s.AbstractConnector] (Thread-1:null) (logid:) Stopped ServerConnector@27808f31{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} 2020-05-17 07:35:15,664 INFO [o.e.j.s.session] (Thread-1:null) (logid:) Stopped scavenging 2020-05-17 07:35:15,665 INFO [o.e.j.s.h.ContextHandler] (Thread-1:null) (logid:) Stopped o.e.j.s.h.MovedContextHandler@d6da883{/,null,UNAVAILABLE} 2020-05-17 07:35:15,669 INFO [o.e.j.s.h.ContextHandler] (Thread-1:null) (logid:) Stopped o.e.j.w.WebAppContext@7bc1a03d{/client,null,UNAVAILABLE}{/usr/share/cloudstack-management/webapp} 2020-05-17 07:35:21,648 INFO [o.a.c.ServerDaemon] (main:null) (logid:) Server configuration file found: /etc/cloudstack/management/server.properties 2020-05-17 07:35:21,655 INFO [o.a.c.ServerDaemon] (main:null) (logid:) Initializing server daemon on null:8080, with https.enabled=false, https.port=8443, context.path=/client 2020-05-17 07:35:21,666 INFO [o.e.j.u.log] (main:null) (logid:) Logging initialized @713ms to org.eclipse.jetty.util.log.Slf4jLog 2020-05-17 07:35:21,857 INFO [o.e.j.s.Server] (main:null) (logid:) jetty-9.4.z-SNAPSHOT, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8 2020-05-17 07:35:21,893 INFO [o.e.j.s.AbstractNCSARequestLog] (main:null) (logid:) Opened /var/log/cloudstack/management/access.log 2020-05-17 07:35:22,007 INFO [o.e.j.w.StandardDescriptorProcessor] (main:null) (logid:) NO JSP Support for /client, did not find org.eclipse.jetty.jsp.JettyJspServlet 2020-05-17 07:35:22,042 INFO [o.e.j.s.session] (main:null) (logid:) DefaultSessionIdManager workerName=node0 2020-05-17 07:35:22,042 INFO [o.e.j.s.session] (main:null) (logid:) No SessionScavenger set, using defaults 2020-05-17 07:35:22,044 INFO [o.e.j.s.session] (main:null) (logid:) Scavenging every 600000ms 2020-05-17 07:35:37,117 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [bootstrap] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context.xml] 2020-05-17 07:35:37,118 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [bootstrap] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2020-05-17 07:35:37,298 DEBUG [c.c.u.c.EncryptionSecretKeyChecker] (main:null) (logid:) Encryption Type: file 2020-05-17 07:35:37,342 INFO [c.c.u.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/management/log4j-cloud.xml 2020-05-17 07:35:37,367 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loaded module context [bootstrap] in 250 ms 2020-05-17 07:35:37,368 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: bootstrap 2020-05-17 07:35:37,368 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: system 2020-05-17 07:35:37,368 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: core 2020-05-17 07:35:37,369 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: allocator 2020-05-17 07:35:37,369 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: host-allocator-random 2020-05-17 07:35:37,369 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: planner 2020-05-17 07:35:37,369 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: api-planner 2020-05-17 07:35:37,369 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: baremetal-planner 2020-05-17 07:35:37,370 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: explicit-dedication 2020-05-17 07:35:37,370 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: host-affinity 2020-05-17 07:35:37,370 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: host-anti-affinity 2020-05-17 07:35:37,370 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: implicit-dedication 2020-05-17 07:35:37,371 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: server-planner 2020-05-17 07:35:37,371 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: skip-heurestics 2020-05-17 07:35:37,371 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: user-concentrated-pod 2020-05-17 07:35:37,371 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: server-allocator 2020-05-17 07:35:37,371 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: api 2020-05-17 07:35:37,371 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: acl-dynamic-role-based 2020-05-17 07:35:37,372 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: acl-static-role-based 2020-05-17 07:35:37,372 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: cloudian 2020-05-17 07:35:37,372 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: ldap 2020-05-17 07:35:37,372 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: md5 2020-05-17 07:35:37,372 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: metrics 2020-05-17 07:35:37,372 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: pbkdf2 2020-05-17 07:35:37,373 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: plaintext 2020-05-17 07:35:37,373 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: prometheus 2020-05-17 07:35:37,373 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: quota 2020-05-17 07:35:37,374 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: rate-limit 2020-05-17 07:35:37,374 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: saml2 2020-05-17 07:35:37,374 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: server-api 2020-05-17 07:35:37,374 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: sha256salted 2020-05-17 07:35:37,374 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: solidfire-intg-test 2020-05-17 07:35:37,375 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: vmware-sioc 2020-05-17 07:35:37,375 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: api-config 2020-05-17 07:35:37,375 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: backend 2020-05-17 07:35:37,375 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: ca 2020-05-17 07:35:37,375 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: root-ca 2020-05-17 07:35:37,375 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: compute 2020-05-17 07:35:37,376 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: baremetal-compute 2020-05-17 07:35:37,376 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: hyperv-compute 2020-05-17 07:35:37,376 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: kvm-compute 2020-05-17 07:35:37,376 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: ovm-compute 2020-05-17 07:35:37,376 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: ovm3-compute 2020-05-17 07:35:37,377 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: server-alert-adapter-compute 2020-05-17 07:35:37,377 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: server-compute 2020-05-17 07:35:37,377 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: server-fencer 2020-05-17 07:35:37,377 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: server-investigator 2020-05-17 07:35:37,378 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: vmware-compute 2020-05-17 07:35:37,378 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: xenserver-compute 2020-05-17 07:35:37,378 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: direct-download 2020-05-17 07:35:37,378 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: network 2020-05-17 07:35:37,379 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: baremetal-network 2020-05-17 07:35:37,379 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: bigswitch 2020-05-17 07:35:37,379 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: cisco-vnmc 2020-05-17 07:35:37,379 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: contrail 2020-05-17 07:35:37,379 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: elb 2020-05-17 07:35:37,379 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: f5 2020-05-17 07:35:37,380 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: globodns 2020-05-17 07:35:37,380 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: netscaler 2020-05-17 07:35:37,380 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: nvp 2020-05-17 07:35:37,380 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: opendaylight 2020-05-17 07:35:37,380 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: ovs 2020-05-17 07:35:37,380 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: paloalto 2020-05-17 07:35:37,381 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: server-network 2020-05-17 07:35:37,381 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: srx 2020-05-17 07:35:37,381 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: ssp 2020-05-17 07:35:37,381 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: vcs 2020-05-17 07:35:37,381 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: vmware-network 2020-05-17 07:35:37,382 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: vsp 2020-05-17 07:35:37,382 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: vxlan 2020-05-17 07:35:37,382 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: server-alert-adapter-backend 2020-05-17 07:35:37,382 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: storage 2020-05-17 07:35:37,382 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: baremetal-storage 2020-05-17 07:35:37,383 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: server-alert-adapter-storage 2020-05-17 07:35:37,383 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: server-storage 2020-05-17 07:35:37,383 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: server-template-adapter 2020-05-17 07:35:37,383 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: storage-allocator 2020-05-17 07:35:37,383 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: storage-image-default 2020-05-17 07:35:37,383 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: storage-image-s3 2020-05-17 07:35:37,383 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: storage-image-swift 2020-05-17 07:35:37,384 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: storage-volume-cloudbyte 2020-05-17 07:35:37,384 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: storage-volume-default 2020-05-17 07:35:37,384 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: storage-volume-solidfire 2020-05-17 07:35:37,384 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: vmware-storage 2020-05-17 07:35:37,384 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: discoverer 2020-05-17 07:35:37,384 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: baremetal-discoverer 2020-05-17 07:35:37,385 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: hyperv-discoverer 2020-05-17 07:35:37,385 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: ovm-discoverer 2020-05-17 07:35:37,385 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: ovm3-discoverer 2020-05-17 07:35:37,385 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: secondary-storage-discoverer 2020-05-17 07:35:37,386 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: server-discoverer 2020-05-17 07:35:37,386 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: vmware-discoverer 2020-05-17 07:35:37,386 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: xenserver-discoverer 2020-05-17 07:35:37,386 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: outofbandmanagement 2020-05-17 07:35:37,386 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: ipmitool 2020-05-17 07:35:37,386 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module Hierarchy: nested-cloudstack 2020-05-17 07:35:37,387 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/system/spring-framework-config-system-context.xml] 2020-05-17 07:35:37,387 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/system/spring-server-system-context.xml] 2020-05-17 07:35:37,388 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/system/spring-core-system-context.xml] 2020-05-17 07:35:37,388 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/system/spring-framework-db-system-context.xml] 2020-05-17 07:35:37,388 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/system/spring-engine-schema-system-checkers-context.xml] 2020-05-17 07:35:37,388 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] 2020-05-17 07:35:37,388 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] 2020-05-17 07:35:37,388 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] 2020-05-17 07:35:37,388 INFO [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] 2020-05-17 07:35:38,357 INFO [c.c.u.d.T.Transaction] (main:null) (logid:) Is Data Base High Availiability enabled? Ans : false 2020-05-17 07:35:38,428 DEBUG [c.c.u.d.DriverLoader] (main:null) (logid:) Successfully loaded DB driver com.mysql.jdbc.Driver 2020-05-17 07:35:38,447 DEBUG [c.c.u.d.DriverLoader] (main:null) (logid:) DB driver com.mysql.jdbc.Driver was already loaded. 2020-05-17 07:35:38,449 DEBUG [c.c.u.d.DriverLoader] (main:null) (logid:) DB driver com.mysql.jdbc.Driver was already loaded. 2020-05-17 07:35:39,037 DEBUG [c.c.u.d.ConnectionConcierge] (main:null) (logid:) Registering a database connection for LockMaster1 2020-05-17 07:35:39,037 INFO [c.c.u.d.Merovingian2] (main:null) (logid:) Cleaning up locks for 121112988680429 2020-05-17 07:35:39,040 INFO [c.c.u.d.Merovingian2] (main:null) (logid:) Released 0 locks for 121112988680429 2020-05-17 07:35:39,387 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null) (logid:) Running system integrity checker com.cloud.upgrade.DatabaseUpgradeChecker@3935e9a8 2020-05-17 07:35:39,389 INFO [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Grabbing lock to check for database upgrade. 2020-05-17 07:35:39,491 DEBUG [c.c.u.d.VersionDaoImpl] (main:null) (logid:) Checking to see if the database is at a version before it was the version table is created 2020-05-17 07:35:39,530 INFO [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) DB version = 4.11.2.0 Code Version = 4.11.3.0 2020-05-17 07:35:39,531 INFO [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Database upgrade must be performed from 4.11.2.0 to 4.11.3.0 2020-05-17 07:35:39,532 DEBUG [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Running upgrade Upgrade41120to41130 to upgrade from 4.11.2.0-4.11.3.0 to 4.11.3.0 2020-05-17 07:35:39,534 DEBUG [c.c.u.d.Upgrade41120to41130] (main:null) (logid:) Updating System Vm template IDs 2020-05-17 07:35:39,538 DEBUG [c.c.u.d.Upgrade41120to41130] (main:null) (logid:) Updating LXC System Vms 2020-05-17 07:35:39,540 WARN [c.c.u.d.Upgrade41120to41130] (main:null) (logid:) 4.11.3.0LXC SystemVm template not found. LXC hypervisor is not used, so not failing upgrade 2020-05-17 07:35:39,543 DEBUG [c.c.u.d.Upgrade41120to41130] (main:null) (logid:) Updating Ovm3 System Vms 2020-05-17 07:35:39,544 WARN [c.c.u.d.Upgrade41120to41130] (main:null) (logid:) 4.11.3.0Ovm3 SystemVm template not found. Ovm3 hypervisor is not used, so not failing upgrade 2020-05-17 07:35:39,545 DEBUG [c.c.u.d.Upgrade41120to41130] (main:null) (logid:) Updating KVM System Vms 2020-05-17 07:35:39,547 ERROR [c.c.u.DatabaseUpgradeChecker] (main:null) (logid:) Unable to upgrade the database com.cloud.utils.exception.CloudRuntimeException: 4.11.3.0KVM SystemVm template not found. Cannot upgrade system Vms at com.cloud.upgrade.dao.Upgrade41120to41130.updateSystemVmTemplates(Upgrade41120to41130.java:203) at com.cloud.upgrade.dao.Upgrade41120to41130.performDataMigration(Upgrade41120to41130.java:60) at com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:586) at com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:666) at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:65) at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:55) at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:183) at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:52) at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:358) at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:159) at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:884) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:552) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:145) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:122) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:245) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:233) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:117) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:79) at org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37) at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:71) at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:58) at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:62) at org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:890) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532) at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:853) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1515) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1477) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:273) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:115) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133) at org.eclipse.jetty.server.Server.start(Server.java:418) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.Server.doStart(Server.java:385) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:200) at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:101) 2020-05-17 07:35:39,550 DEBUG [c.c.u.d.T.Transaction] (main:null) (logid:) Rolling back the transaction: Time = 18 Name = Upgrade; called by -TransactionLegacy.rollback:890-TransactionLegacy.removeUpTo:833-TransactionLegacy.close:657-DatabaseUpgradeChecker.upgrade:597-DatabaseUpgradeChecker.check:666-CloudStackExtendedLifeCycle.checkIntegrity:65-CloudStackExtendedLifeCycle.start:55-DefaultLifecycleProcessor.doStart:183-DefaultLifecycleProcessor.access$200:52-DefaultLifecycleProcessor$LifecycleGroup.start:358-DefaultLifecycleProcessor.startBeans:159-DefaultLifecycleProcessor.onRefresh:123 2020-05-17 07:35:39,604 WARN [o.a.c.s.m.c.ResourceApplicationContext] (main:null) (logid:) Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'cloudStackLifeCycle'; nested exception is com.cloud.utils.exception.CloudRuntimeException: Unable to upgrade the database 2020-05-17 07:35:39,605 WARN [o.e.j.w.WebAppContext] (main:null) (logid:) Failed startup of context o.e.j.w.WebAppContext@7bc1a03d{/client,file:///usr/share/cloudstack-management/webapp/,UNAVAILABLE}{/usr/share/cloudstack-management/webapp} org.springframework.context.ApplicationContextException: Failed to start bean 'cloudStackLifeCycle'; nested exception is com.cloud.utils.exception.CloudRuntimeException: Unable to upgrade the database at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:186) at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:52) at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:358) at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:159) at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:884) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:552) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:145) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:122) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:245) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:233) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:117) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:79) at org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37) at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:71) at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:58) at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:62) at org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:890) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532) at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:853) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1515) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1477) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:273) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:115) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133) at org.eclipse.jetty.server.Server.start(Server.java:418) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.Server.doStart(Server.java:385) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:200) at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:101) Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to upgrade the database at com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:595) at com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:666) at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:65) ... 47 more Caused by: com.cloud.utils.exception.CloudRuntimeException: 4.11.3.0KVM SystemVm template not found. Cannot upgrade system Vms at com.cloud.upgrade.dao.Upgrade41120to41130.updateSystemVmTemplates(Upgrade41120to41130.java:203) at com.cloud.upgrade.dao.Upgrade41120to41130.performDataMigration(Upgrade41120to41130.java:60) at com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:586) ... 49 more 2020-05-17 07:35:39,609 INFO [o.e.j.s.h.ContextHandler] (main:null) (logid:) Started o.e.j.s.h.MovedContextHandler@d6da883{/,null,AVAILABLE} 2020-05-17 07:35:39,658 INFO [o.e.j.s.AbstractConnector] (main:null) (logid:) Started ServerConnector@27808f31{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} 2020-05-17 07:35:39,659 INFO [o.e.j.s.Server] (main:null) (logid:) Started @18708ms
