OK, got it, digging in... Thanks everyone, David
David Merrill Senior Systems Engineer, Managed and Private/Hybrid Cloud Services OTELCO 92 Oak Street, Portland ME 04101 office 207.772.5678 <callto:207.772.5678> http://www.otelco.com/cloud-and-managed-services Confidentiality Message The information contained in this e-mail transmission may be confidential and legally privileged. If you are not the intended recipient, you are notified that any dissemination, distribution, copying or other use of this information, including attachments, is prohibited. If you received this message in error, please call me at 207.772.5678 <callto:207.772.5678> so this error can be corrected. On 5/18/20, 7:44 AM, "Andrija Panic" <andrija.pa...@gmail.com> wrote: David, the procedure you laid out is correct - rollback DB, downgrade, etc As Luis mentioned, make sure to also rollback the "mysql-connector-java" in case it was upgraded (if you left the mysql repo enabled during the upgrade). There are ways to hack the DB, vm_template table and also use the "cloud-install-sys-tmplt"... but a clean rollback (as it's very easy in your test env) is a much better way to proceed. Cheers Andrija On Mon, 18 May 2020 at 10:06, Richard Lawley <rich...@richardlawley.com> wrote: > The database upgrade does not happen unless the systemVM templates have > been added, so nothing non-reversible has happened yet. You can just use > yum to downgrade to 4.11.2 and you'll be fine (we've also accidentally done > this at some point!). > > I'd recommend disabling your cloudstack yum repo so that it doesn't happen > again. > > On Sun, 17 May 2020 at 21:01, Luis <lmartinez...@yahoo.com.invalid> wrote: > > > Dont forget to downgrade the database connector or it will not work > > > > Sent from Yahoo Mail on Android > > > > On Sun, May 17, 2020 at 3:47 PM, David Merrill< > david.merr...@otelco.com> > > wrote: Hi All, > > > > I've got a CloudStack 4.11.2 lab running on a single host & made the > > (dumb) mistake of running OS updates without updating the system VMs > > beforehand. The CloudStack packages upgraded to 4.11.3 just fine but now > > CloudStack management services wont start (see below). > > > > Here's my question: > > > > - Is it *at all* possible to rectify this (now, post package updates) by > > getting the new system templates in via the CLI tools > > > (/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt)? > > - The error on the logs *seems* simple, if management could find the new > > template maybe things could proceed? > > > > My impression is no given research I've done so far which seems to amount > > to: > > > > - roll back to 4.11.2 > > - use the pre-upgrade dump of the DB (which I made) > > - start management services > > - get the new system VMs properly (like I should have in the first > place) > > - upgrade to 4.11.3 > > > > Humbly (& slightly embarrassed) yours, and thanks, > > David > > > > David Merrill > > Senior Systems Engineer, > > Managed and Private/Hybrid Cloud Services > > OTELCO > > 92 Oak Street, Portland ME 04101 > > office 207.772.5678 <callto:207.772.5678> > > http://www.otelco.com/cloud-and-managed-services > > > > Confidentiality Message > > The information contained in this e-mail transmission may be confidential > > and legally privileged. If you are not the intended recipient, you are > > notified that any dissemination, distribution, copying or other use of > this > > information, including attachments, is prohibited. If you received this > > message in error, please call me at 207.772.5678 <callto:207.772.5678> so > > this error can be corrected. > > > > > > On 5/17/20, 8:05 AM, "engineer...@reliablenetworks.com" < > > engineer...@reliablenetworks.com> wrote: > > > > 2020-05-17 07:35:15,659 INFO [o.e.j.s.AbstractConnector] > > (Thread-1:null) (logid:) Stopped ServerConnector@27808f31 > > {HTTP/1.1,[http/1.1]}{0.0.0.0:8080} > > 2020-05-17 07:35:15,664 INFO [o.e.j.s.session] (Thread-1:null) > > (logid:) Stopped scavenging > > 2020-05-17 07:35:15,665 INFO [o.e.j.s.h.ContextHandler] > > (Thread-1:null) (logid:) Stopped o.e.j.s.h.MovedContextHandler@d6da883 > > {/,null,UNAVAILABLE} > > 2020-05-17 07:35:15,669 INFO [o.e.j.s.h.ContextHandler] > > (Thread-1:null) (logid:) Stopped o.e.j.w.WebAppContext@7bc1a03d > > {/client,null,UNAVAILABLE}{/usr/share/cloudstack-management/webapp} > > 2020-05-17 07:35:21,648 INFO [o.a.c.ServerDaemon] (main:null) > > (logid:) Server configuration file found: > > /etc/cloudstack/management/server.properties > > 2020-05-17 07:35:21,655 INFO [o.a.c.ServerDaemon] (main:null) > > (logid:) Initializing server daemon on null:8080, with > https.enabled=false, > > https.port=8443, context.path=/client > > 2020-05-17 07:35:21,666 INFO [o.e.j.u.log] (main:null) (logid:) > > Logging initialized @713ms to org.eclipse.jetty.util.log.Slf4jLog > > 2020-05-17 07:35:21,857 INFO [o.e.j.s.Server] (main:null) (logid:) > > jetty-9.4.z-SNAPSHOT, build timestamp: 2017-11-21T16:27:37-05:00, git > hash: > > 82b8fb23f757335bb3329d540ce37a2a2615f0a8 > > 2020-05-17 07:35:21,893 INFO [o.e.j.s.AbstractNCSARequestLog] > > (main:null) (logid:) Opened /var/log/cloudstack/management/access.log > > 2020-05-17 07:35:22,007 INFO [o.e.j.w.StandardDescriptorProcessor] > > (main:null) (logid:) NO JSP Support for /client, did not find > > org.eclipse.jetty.jsp.JettyJspServlet > > 2020-05-17 07:35:22,042 INFO [o.e.j.s.session] (main:null) (logid:) > > DefaultSessionIdManager workerName=node0 > > 2020-05-17 07:35:22,042 INFO [o.e.j.s.session] (main:null) (logid:) > > No SessionScavenger set, using defaults > > 2020-05-17 07:35:22,044 INFO [o.e.j.s.session] (main:null) (logid:) > > Scavenging every 600000ms > > 2020-05-17 07:35:37,117 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading > > module context [bootstrap] from URL > > > [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context.xml] > > 2020-05-17 07:35:37,118 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading > > module context [bootstrap] from URL > > > [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] > > 2020-05-17 07:35:37,298 DEBUG [c.c.u.c.EncryptionSecretKeyChecker] > > (main:null) (logid:) Encryption Type: file > > 2020-05-17 07:35:37,342 INFO [c.c.u.LogUtils] (main:null) (logid:) > > log4j configuration found at /etc/cloudstack/management/log4j-cloud.xml > > 2020-05-17 07:35:37,367 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loaded > > module context [bootstrap] in 250 ms > > 2020-05-17 07:35:37,368 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: bootstrap > > 2020-05-17 07:35:37,368 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: system > > 2020-05-17 07:35:37,368 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: core > > 2020-05-17 07:35:37,369 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: allocator > > 2020-05-17 07:35:37,369 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: host-allocator-random > > 2020-05-17 07:35:37,369 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: planner > > 2020-05-17 07:35:37,369 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: api-planner > > 2020-05-17 07:35:37,369 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: baremetal-planner > > 2020-05-17 07:35:37,370 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: explicit-dedication > > 2020-05-17 07:35:37,370 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: host-affinity > > 2020-05-17 07:35:37,370 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: host-anti-affinity > > 2020-05-17 07:35:37,370 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: implicit-dedication > > 2020-05-17 07:35:37,371 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: server-planner > > 2020-05-17 07:35:37,371 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: skip-heurestics > > 2020-05-17 07:35:37,371 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: user-concentrated-pod > > 2020-05-17 07:35:37,371 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: server-allocator > > 2020-05-17 07:35:37,371 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: api > > 2020-05-17 07:35:37,371 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: acl-dynamic-role-based > > 2020-05-17 07:35:37,372 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: acl-static-role-based > > 2020-05-17 07:35:37,372 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: cloudian > > 2020-05-17 07:35:37,372 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: ldap > > 2020-05-17 07:35:37,372 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: md5 > > 2020-05-17 07:35:37,372 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: metrics > > 2020-05-17 07:35:37,372 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: pbkdf2 > > 2020-05-17 07:35:37,373 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: plaintext > > 2020-05-17 07:35:37,373 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: prometheus > > 2020-05-17 07:35:37,373 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: quota > > 2020-05-17 07:35:37,374 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: rate-limit > > 2020-05-17 07:35:37,374 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: saml2 > > 2020-05-17 07:35:37,374 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: server-api > > 2020-05-17 07:35:37,374 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: sha256salted > > 2020-05-17 07:35:37,374 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: solidfire-intg-test > > 2020-05-17 07:35:37,375 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: vmware-sioc > > 2020-05-17 07:35:37,375 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: api-config > > 2020-05-17 07:35:37,375 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: backend > > 2020-05-17 07:35:37,375 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: ca > > 2020-05-17 07:35:37,375 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: root-ca > > 2020-05-17 07:35:37,375 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: compute > > 2020-05-17 07:35:37,376 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: baremetal-compute > > 2020-05-17 07:35:37,376 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: hyperv-compute > > 2020-05-17 07:35:37,376 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: kvm-compute > > 2020-05-17 07:35:37,376 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: ovm-compute > > 2020-05-17 07:35:37,376 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: ovm3-compute > > 2020-05-17 07:35:37,377 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: server-alert-adapter-compute > > 2020-05-17 07:35:37,377 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: server-compute > > 2020-05-17 07:35:37,377 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: server-fencer > > 2020-05-17 07:35:37,377 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: server-investigator > > 2020-05-17 07:35:37,378 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: vmware-compute > > 2020-05-17 07:35:37,378 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: xenserver-compute > > 2020-05-17 07:35:37,378 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: direct-download > > 2020-05-17 07:35:37,378 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: network > > 2020-05-17 07:35:37,379 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: baremetal-network > > 2020-05-17 07:35:37,379 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: bigswitch > > 2020-05-17 07:35:37,379 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: cisco-vnmc > > 2020-05-17 07:35:37,379 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: contrail > > 2020-05-17 07:35:37,379 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: elb > > 2020-05-17 07:35:37,379 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: f5 > > 2020-05-17 07:35:37,380 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: globodns > > 2020-05-17 07:35:37,380 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: netscaler > > 2020-05-17 07:35:37,380 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: nvp > > 2020-05-17 07:35:37,380 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: opendaylight > > 2020-05-17 07:35:37,380 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: ovs > > 2020-05-17 07:35:37,380 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: paloalto > > 2020-05-17 07:35:37,381 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: server-network > > 2020-05-17 07:35:37,381 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: srx > > 2020-05-17 07:35:37,381 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: ssp > > 2020-05-17 07:35:37,381 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: vcs > > 2020-05-17 07:35:37,381 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: vmware-network > > 2020-05-17 07:35:37,382 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: vsp > > 2020-05-17 07:35:37,382 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: vxlan > > 2020-05-17 07:35:37,382 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: server-alert-adapter-backend > > 2020-05-17 07:35:37,382 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: storage > > 2020-05-17 07:35:37,382 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: baremetal-storage > > 2020-05-17 07:35:37,383 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: server-alert-adapter-storage > > 2020-05-17 07:35:37,383 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: server-storage > > 2020-05-17 07:35:37,383 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: server-template-adapter > > 2020-05-17 07:35:37,383 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: storage-allocator > > 2020-05-17 07:35:37,383 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: storage-image-default > > 2020-05-17 07:35:37,383 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: storage-image-s3 > > 2020-05-17 07:35:37,383 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: storage-image-swift > > 2020-05-17 07:35:37,384 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: storage-volume-cloudbyte > > 2020-05-17 07:35:37,384 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: storage-volume-default > > 2020-05-17 07:35:37,384 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: storage-volume-solidfire > > 2020-05-17 07:35:37,384 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: vmware-storage > > 2020-05-17 07:35:37,384 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: discoverer > > 2020-05-17 07:35:37,384 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: baremetal-discoverer > > 2020-05-17 07:35:37,385 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: hyperv-discoverer > > 2020-05-17 07:35:37,385 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: ovm-discoverer > > 2020-05-17 07:35:37,385 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: ovm3-discoverer > > 2020-05-17 07:35:37,385 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: secondary-storage-discoverer > > 2020-05-17 07:35:37,386 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: server-discoverer > > 2020-05-17 07:35:37,386 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: vmware-discoverer > > 2020-05-17 07:35:37,386 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: xenserver-discoverer > > 2020-05-17 07:35:37,386 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: outofbandmanagement > > 2020-05-17 07:35:37,386 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: ipmitool > > 2020-05-17 07:35:37,386 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Module > > Hierarchy: nested-cloudstack > > 2020-05-17 07:35:37,387 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading > > module context [system] from URL > > > [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/system/spring-framework-config-system-context.xml] > > 2020-05-17 07:35:37,387 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading > > module context [system] from URL > > > [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/system/spring-server-system-context.xml] > > 2020-05-17 07:35:37,388 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading > > module context [system] from URL > > > [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/system/spring-core-system-context.xml] > > 2020-05-17 07:35:37,388 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading > > module context [system] from URL > > > [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/system/spring-framework-db-system-context.xml] > > 2020-05-17 07:35:37,388 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading > > module context [system] from URL > > > [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/system/spring-engine-schema-system-checkers-context.xml] > > 2020-05-17 07:35:37,388 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading > > module context [system] from URL > > > [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml] > > 2020-05-17 07:35:37,388 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading > > module context [system] from URL > > > [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml] > > 2020-05-17 07:35:37,388 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading > > module context [system] from URL > > > [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml] > > 2020-05-17 07:35:37,388 INFO > > [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading > > module context [system] from URL > > > [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.11.3.0.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml] > > 2020-05-17 07:35:38,357 INFO [c.c.u.d.T.Transaction] (main:null) > > (logid:) Is Data Base High Availiability enabled? Ans : false > > 2020-05-17 07:35:38,428 DEBUG [c.c.u.d.DriverLoader] (main:null) > > (logid:) Successfully loaded DB driver com.mysql.jdbc.Driver > > 2020-05-17 07:35:38,447 DEBUG [c.c.u.d.DriverLoader] (main:null) > > (logid:) DB driver com.mysql.jdbc.Driver was already loaded. > > 2020-05-17 07:35:38,449 DEBUG [c.c.u.d.DriverLoader] (main:null) > > (logid:) DB driver com.mysql.jdbc.Driver was already loaded. > > 2020-05-17 07:35:39,037 DEBUG [c.c.u.d.ConnectionConcierge] > > (main:null) (logid:) Registering a database connection for LockMaster1 > > 2020-05-17 07:35:39,037 INFO [c.c.u.d.Merovingian2] (main:null) > > (logid:) Cleaning up locks for 121112988680429 > > 2020-05-17 07:35:39,040 INFO [c.c.u.d.Merovingian2] (main:null) > > (logid:) Released 0 locks for 121112988680429 > > 2020-05-17 07:35:39,387 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] > > (main:null) (logid:) Running system integrity checker > > com.cloud.upgrade.DatabaseUpgradeChecker@3935e9a8 > > 2020-05-17 07:35:39,389 INFO [c.c.u.DatabaseUpgradeChecker] > > (main:null) (logid:) Grabbing lock to check for database upgrade. > > 2020-05-17 07:35:39,491 DEBUG [c.c.u.d.VersionDaoImpl] (main:null) > > (logid:) Checking to see if the database is at a version before it was > the > > version table is created > > 2020-05-17 07:35:39,530 INFO [c.c.u.DatabaseUpgradeChecker] > > (main:null) (logid:) DB version = 4.11.2.0 Code Version = 4.11.3.0 > > 2020-05-17 07:35:39,531 INFO [c.c.u.DatabaseUpgradeChecker] > > (main:null) (logid:) Database upgrade must be performed from 4.11.2.0 to > > 4.11.3.0 > > 2020-05-17 07:35:39,532 DEBUG [c.c.u.DatabaseUpgradeChecker] > > (main:null) (logid:) Running upgrade Upgrade41120to41130 to upgrade from > > 4.11.2.0-4.11.3.0 to 4.11.3.0 > > 2020-05-17 07:35:39,534 DEBUG [c.c.u.d.Upgrade41120to41130] > > (main:null) (logid:) Updating System Vm template IDs > > 2020-05-17 07:35:39,538 DEBUG [c.c.u.d.Upgrade41120to41130] > > (main:null) (logid:) Updating LXC System Vms > > 2020-05-17 07:35:39,540 WARN [c.c.u.d.Upgrade41120to41130] > > (main:null) (logid:) 4.11.3.0LXC SystemVm template not found. LXC > > hypervisor is not used, so not failing upgrade > > 2020-05-17 07:35:39,543 DEBUG [c.c.u.d.Upgrade41120to41130] > > (main:null) (logid:) Updating Ovm3 System Vms > > 2020-05-17 07:35:39,544 WARN [c.c.u.d.Upgrade41120to41130] > > (main:null) (logid:) 4.11.3.0Ovm3 SystemVm template not found. Ovm3 > > hypervisor is not used, so not failing upgrade > > 2020-05-17 07:35:39,545 DEBUG [c.c.u.d.Upgrade41120to41130] > > (main:null) (logid:) Updating KVM System Vms > > 2020-05-17 07:35:39,547 ERROR [c.c.u.DatabaseUpgradeChecker] > > (main:null) (logid:) Unable to upgrade the database > > com.cloud.utils.exception.CloudRuntimeException: 4.11.3.0KVM SystemVm > > template not found. Cannot upgrade system Vms > > at > > > com.cloud.upgrade.dao.Upgrade41120to41130.updateSystemVmTemplates(Upgrade41120to41130.java:203) > > at > > > com.cloud.upgrade.dao.Upgrade41120to41130.performDataMigration(Upgrade41120to41130.java:60) > > at > > > com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:586) > > at > > > com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:666) > > at > > > org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:65) > > at > > > org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:55) > > at > > > org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:183) > > at > > > org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:52) > > at > > > org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:358) > > at > > > org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:159) > > at > > > org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) > > at > > > org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:884) > > at > > > org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:552) > > at > > > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:145) > > at > > > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:122) > > at > > > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:245) > > at > > > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250) > > at > > > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:233) > > at > > > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:117) > > at > > > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:79) > > at > > > org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37) > > at > > > org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:71) > > at > > > org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:58) > > at > > > org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:62) > > at > > > org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52) > > at > > > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:890) > > at > > > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532) > > at > > > org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:853) > > at > > > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344) > > at > > > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1515) > > at > > > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1477) > > at > > > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785) > > at > > > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261) > > at > > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545) > > at > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > > at > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133) > > at > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107) > > at > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) > > at > > > org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:273) > > at > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > > at > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133) > > at > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:115) > > at > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) > > at > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > > at > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133) > > at org.eclipse.jetty.server.Server.start(Server.java:418) > > at > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107) > > at > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) > > at org.eclipse.jetty.server.Server.doStart(Server.java:385) > > at > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > > at > org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:200) > > at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:101) > > 2020-05-17 07:35:39,550 DEBUG [c.c.u.d.T.Transaction] (main:null) > > (logid:) Rolling back the transaction: Time = 18 Name = Upgrade; called > by > > > -TransactionLegacy.rollback:890-TransactionLegacy.removeUpTo:833-TransactionLegacy.close:657-DatabaseUpgradeChecker.upgrade:597-DatabaseUpgradeChecker.check:666-CloudStackExtendedLifeCycle.checkIntegrity:65-CloudStackExtendedLifeCycle.start:55-DefaultLifecycleProcessor.doStart:183-DefaultLifecycleProcessor.access$200:52-DefaultLifecycleProcessor$LifecycleGroup.start:358-DefaultLifecycleProcessor.startBeans:159-DefaultLifecycleProcessor.onRefresh:123 > > 2020-05-17 07:35:39,604 WARN > [o.a.c.s.m.c.ResourceApplicationContext] > > (main:null) (logid:) Exception encountered during context initialization > - > > cancelling refresh attempt: > > org.springframework.context.ApplicationContextException: Failed to start > > bean 'cloudStackLifeCycle'; nested exception is > > com.cloud.utils.exception.CloudRuntimeException: Unable to upgrade the > > database > > 2020-05-17 07:35:39,605 WARN [o.e.j.w.WebAppContext] (main:null) > > (logid:) Failed startup of context o.e.j.w.WebAppContext@7bc1a03d > > > {/client,file:///usr/share/cloudstack-management/webapp/,UNAVAILABLE}{/usr/share/cloudstack-management/webapp} > > org.springframework.context.ApplicationContextException: Failed to > > start bean 'cloudStackLifeCycle'; nested exception is > > com.cloud.utils.exception.CloudRuntimeException: Unable to upgrade the > > database > > at > > > org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:186) > > at > > > org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:52) > > at > > > org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:358) > > at > > > org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:159) > > at > > > org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) > > at > > > org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:884) > > at > > > org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:552) > > at > > > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:145) > > at > > > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:122) > > at > > > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:245) > > at > > > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250) > > at > > > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:233) > > at > > > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:117) > > at > > > org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:79) > > at > > > org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37) > > at > > > org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:71) > > at > > > org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:58) > > at > > > org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:62) > > at > > > org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52) > > at > > > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:890) > > at > > > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532) > > at > > > org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:853) > > at > > > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344) > > at > > > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1515) > > at > > > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1477) > > at > > > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785) > > at > > > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261) > > at > > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545) > > at > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > > at > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133) > > at > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107) > > at > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) > > at > > > org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:273) > > at > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > > at > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133) > > at > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:115) > > at > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) > > at > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > > at > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133) > > at org.eclipse.jetty.server.Server.start(Server.java:418) > > at > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107) > > at > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) > > at org.eclipse.jetty.server.Server.doStart(Server.java:385) > > at > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > > at > org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:200) > > at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:101) > > Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to > > upgrade the database > > at > > > com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:595) > > at > > > com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:666) > > at > > > org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:65) > > ... 47 more > > Caused by: com.cloud.utils.exception.CloudRuntimeException: > > 4.11.3.0KVM SystemVm template not found. Cannot upgrade system Vms > > at > > > com.cloud.upgrade.dao.Upgrade41120to41130.updateSystemVmTemplates(Upgrade41120to41130.java:203) > > at > > > com.cloud.upgrade.dao.Upgrade41120to41130.performDataMigration(Upgrade41120to41130.java:60) > > at > > > com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:586) > > ... 49 more > > 2020-05-17 07:35:39,609 INFO [o.e.j.s.h.ContextHandler] (main:null) > > (logid:) Started o.e.j.s.h.MovedContextHandler@d6da883{/,null,AVAILABLE} > > 2020-05-17 07:35:39,658 INFO [o.e.j.s.AbstractConnector] (main:null) > > (logid:) Started ServerConnector@27808f31{HTTP/1.1,[http/1.1]}{ > > 0.0.0.0:8080} > > 2020-05-17 07:35:39,659 INFO [o.e.j.s.Server] (main:null) (logid:) > > Started @18708ms > > > > > > > > > -- Andrija Panić
