Re: [VOTE] Apache Cloudstack 4.15.0.0 and UI [RC3]
Hi Rohit and Andrija, Thanks for reply.I will try again drop mysql 5.7 and reinstall back mysql8 , All is working now after revert to mysql 5.7 and restore the backup . On Mon, Dec 28, 2020 at 6:39 PM Rohit Yadav wrote: > Hi Hean, > > I think you've figured out the first issue to be missing systemvmtemplate, > the upgrade requires that the 4.15 systemvmtemplate is registered prior to > the upgrade. > > MySQL8 should work with 4.15 RC3 (as Andrija notes), however, if you're > doing an in-place upgrade you may want to backup the DB dumps, stop all old > CloudStack services (mgmt and usage servers), and then try to upgrade MySQL > 5.x to 8 and then follow the upgrade. You may also take DB backups and > restore them in a new MySQL8 instance. > > The do-release-upgrade would perform an in-place upgrade of Ubuntu 18.04 > to 20.04 installation which may have issues of its own, can you try a fresh > installation of Ubuntu 20.04 + MySQL8 and try again? Thanks. > > > Regards. > > > From: Andrija Panic > Sent: Monday, December 28, 2020 14:04 > To: users > Subject: Re: [VOTE] Apache Cloudstack 4.15.0.0 and UI [RC3] > > MySQL 8 should work - support for it was introduced in 4.15, if not > mistaken (5.7 is still a safe bet). > > Your issue seems to be an unclean restore of the DB, based on the input > you've shared. You need to drop your cloud/cloud_usage DBs, create empty > ones, restore them both from backup, start the old mgmt server, make sure > that the proper systemVM template is registered with the EXACT name as > specified in the upgrade guide (you have to use that name, otherwise later > the DB upgrade will fail) and only then proceed with the upgrade to 4.15. > > > rohit.ya...@shapeblue.com > www.shapeblue.com > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > @shapeblue > > > > On Mon, 28 Dec 2020 at 06:33, Hean Seng wrote: > > > Just for update, I tested on Ubuntu 18, and upgrade to ACS5.15, no > issue . > > > > After that MGMT, do-release-upgrade to Ubuntu 20, and no issue on upgrade > > to Ubuntu20, However MySQL need to downgrade to MySQL 5.7. and restore > back > > the DB . Not sure how to make it work on MySQL8 yet. > > > > Create VM, Snapshot, Delivete, those is no issue as well. > > > > > > > > On Sun, Dec 27, 2020 at 1:08 AM Hean Seng wrote: > > > > > I think the main issue is the first time not recognize the systemvm , > > > although already install they SystemVM > > > > > > I do bare new installation on 4.15, CentOS7,KVM, , and it works > > > > > > On Sun, Dec 27, 2020 at 12:31 AM Sergey Levitskiy > > > > wrote: > > > > > >> You can try this. Restore your DB backup, register SSVM template and > run > > >> the following against your MySQL DB before starting the upgrade. > > >> > > >> ALTER TABLE `cloud`.`project_account` > > >> ADD CONSTRAINT `fk_project_account__account_id` FOREIGN > > >> KEY(`account_id`) REFERENCES `account`(`id`) ON DELETE CASCADE , > > >> ADD CONSTRAINT `uc_project_account__project_id_account_id_user_id` > > >> UNIQUE (`project_id`, `account_id`, `user_id`) ; > > >> > > >> > > >> If it still fails capture and post full management server log. > > >> > > >> > > >> Thanks, > > >> Sergey > > >> > > >> On 12/26/20, 2:27 AM, "Hean Seng" wrote: > > >> > > >> I restore the backup db, and reregister the system template using > > >> cloud-install-sys-tmplt > > >> , it sill getting error. > > >> > > >> stemVm template not found. Ovm3 hypervisor is not used, so not > > failing > > >> upgrade > > >> > > >> 2020-12-26 10:11:37,713 DEBUG [c.c.u.d.Upgrade41400to41500] > > >> (main:null) > > >> (logid:) Updating KVM System Vms > > >> > > >> 2020-12-26 10:11:37,720 ERROR [c.c.u.DatabaseUpgradeChecker] > > >> (main:null) > > >> (logid:) Unable to upgrade the database > > >> > > >> com.cloud.utils.exception.CloudRuntimeException: 4.15.0.0KVM > > SystemVm > > >> template not found. Cannot upgrade system Vms > > >> > > >> at > > >> > > >> > > > com.cloud.upgrade.dao.Upgrade41400to41500.updateSystemVmTemplates(Upgrade41400to41500.java:214) > > >> > > >> at > > >> > > >> > > > com.cloud.upgrade.dao.Upgrade41400to41500.performDataMigration(Upgrade41400to41500.java:70) > > >> > > >> On Sat, Dec 26, 2020 at 5:48 PM Hean Seng > > wrote: > > >> > > >> > For first time I upgrade and start the MGMT server , it show > > >> > following error: > > >> > > > >> > 2020-12-26 09:02:32,499 DEBUG [c.c.u.d.Upgrade41400to41500] > > >> (main:null) > > >> > (logid:) Updating System Vm template IDs > > >> > > > >> > 2020-12-26 09:02:32,503 DEBUG [c.c.u.d.Upgrade41400to41500] > > >> (main:null) > > >> > (logid:) Updating KVM System Vms > > >> > > > >> > 2020-12-26 09:02:32,511 ERROR [c.c.u.DatabaseUpgradeChecker] > > >> (main:null) > > >> > (logid:) Unable to upgrade the database > > >> > > > >> >
Re: SSVM and CPVM agent unable to start after console proxy SSL certificate update
Hi, Can you try to manually start the cloud service, for example: "service cloud start" and tail/share the logs which may explain why the java process is not running. If that does not work, you may also try to validate/verify the certificates (including any chain/intermediate certificates) you've uploaded and destroy the old CPVM/SSVM. For more information on SSL certificate setup, you may read this 4.11-specific blog https://www.shapeblue.com/securing-cloudstack-4-11-with-https-tls/ which I think is applicable for 4.9 as well. Regards. From: Cloud List Sent: Saturday, December 26, 2020 09:42 To: users@cloudstack.apache.org ; dev Subject: SSVM and CPVM agent unable to start after console proxy SSL certificate update Hi, Merry Christmas to all. We are using Cloudstack with KVM hypervisor. Since our console proxy SSL certificate has expired, we updated our new SSL certificate using below method: http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/systemvm.html#using-a-ssl-certificate-for-the-console-proxy We have done the above method in the past years without any issues, however this time round, both the SSVM and CPVM agents are not able to start after the update. The state for both VMs are up but agents are in "disconnected" state. We are still able to login to the SSVM, and found out that the cloud service is not running. root@s-4200-VM:~# service cloud status CloudStack cloud service is not running Tried to start the service: root@s-4200-VM:~# service cloud start Starting CloudStack cloud service (type=secstorage) Success But the service is not started: root@s-4200-VM:~# service cloud status CloudStack cloud service is not running Below is the logs from /var/log/cloud.log: = Sat Dec 26 03:45:04 UTC 2020 Executing cloud-early-config Sat Dec 26 03:45:04 UTC 2020 Detected that we are running inside kvm guest Sat Dec 26 03:45:04 UTC 2020 Found a non empty cmdline file. Will now exit the loop and proceed with configuration. Sat Dec 26 03:45:04 UTC 2020 Patching cloud service Sat Dec 26 03:45:10 UTC 2020 Updating log4j-cloud.xml Sat Dec 26 03:45:10 UTC 2020 Setting up secondary storage system vm Sat Dec 26 03:45:10 UTC 2020 checking that eth0 has IP Sat Dec 26 03:45:11 UTC 2020 waiting for eth0 interface setup with ip timer=0 Sat Dec 26 03:45:11 UTC 2020 checking that eth1 has IP Sat Dec 26 03:45:11 UTC 2020 checking that eth2 has IP Sat Dec 26 03:45:20 UTC 2020 checking that eth3 has IP Sat Dec 26 03:45:20 UTC 2020 Successfully setup storage network with STORAGE_IP:10.19.22.67, STORAGE_NETMASK:255.255.240.0, STORAGE_CIDR: Sat Dec 26 03:45:20 UTC 2020 Setting up route of RFC1918 space to 10.19.16.1 Sat Dec 26 03:45:20 UTC 2020 Setting up apache web server Sat Dec 26 03:45:20 UTC 2020 setting up apache2 for post upload of volume/template Sat Dec 26 03:45:20 UTC 2020 rewrite rules already exist in file /etc/apache2/sites-available/default-ssl Sat Dec 26 03:45:20 UTC 2020 adding cors rules to file: /etc/apache2/sites-available/default-ssl Sat Dec 26 03:45:21 UTC 2020 cloud: disable rp_filter Sat Dec 26 03:45:21 UTC 2020 disable rpfilter Sat Dec 26 03:45:21 UTC 2020 cloud: enable_fwding = 0 Sat Dec 26 03:45:21 UTC 2020 enable_fwding = 0 Sat Dec 26 03:45:21 UTC 2020 Enable service haproxy = 0 Sat Dec 26 03:45:21 UTC 2020 Processors = 1 Enable service = 0 Sat Dec 26 03:45:21 UTC 2020 Enable service dnsmasq = 0 Sat Dec 26 03:45:21 UTC 2020 Enable service cloud-passwd-srvr = 0 Sat Dec 26 03:45:21 UTC 2020 Enable service cloud = 1 = Result of /usr/local/cloud/systemvm/ssvm-check.sh: = root@s-4200-VM:/var/log# /usr/local/cloud/systemvm/ssvm-check.sh First DNS server is 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 48 data bytes 56 bytes from 8.8.8.8: icmp_seq=0 ttl=122 time=0.531 ms 56 bytes from 8.8.8.8: icmp_seq=1 ttl=122 time=0.676 ms --- 8.8.8.8 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.531/0.604/0.676/0.073 ms Good: Can ping DNS server Good: DNS resolves download.cloud.com ERROR: NFS is not currently mounted Try manually mounting from inside the VM NFS server is X.X.201.1 PING X.X.201.1 (X.X.201.1): 48 data bytes 56 bytes from X.X.201.1: icmp_seq=0 ttl=255 time=0.463 ms 56 bytes from X.X.201.1: icmp_seq=1 ttl=255 time=0.482 ms --- X.X.201.1 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.463/0.473/0.482/0.000 ms Good: Can ping nfs server Management server is 10.237.3.8. Checking connectivity. Good: Can connect to management server port 8250 ERROR: Java process not running. Try restarting the SSVM. root@s-4200-VM:/var/log# = The result is OK except
Re: [VOTE] Apache Cloudstack 4.15.0.0 and UI [RC3]
Hi Hean, I think you've figured out the first issue to be missing systemvmtemplate, the upgrade requires that the 4.15 systemvmtemplate is registered prior to the upgrade. MySQL8 should work with 4.15 RC3 (as Andrija notes), however, if you're doing an in-place upgrade you may want to backup the DB dumps, stop all old CloudStack services (mgmt and usage servers), and then try to upgrade MySQL 5.x to 8 and then follow the upgrade. You may also take DB backups and restore them in a new MySQL8 instance. The do-release-upgrade would perform an in-place upgrade of Ubuntu 18.04 to 20.04 installation which may have issues of its own, can you try a fresh installation of Ubuntu 20.04 + MySQL8 and try again? Thanks. Regards. From: Andrija Panic Sent: Monday, December 28, 2020 14:04 To: users Subject: Re: [VOTE] Apache Cloudstack 4.15.0.0 and UI [RC3] MySQL 8 should work - support for it was introduced in 4.15, if not mistaken (5.7 is still a safe bet). Your issue seems to be an unclean restore of the DB, based on the input you've shared. You need to drop your cloud/cloud_usage DBs, create empty ones, restore them both from backup, start the old mgmt server, make sure that the proper systemVM template is registered with the EXACT name as specified in the upgrade guide (you have to use that name, otherwise later the DB upgrade will fail) and only then proceed with the upgrade to 4.15. rohit.ya...@shapeblue.com www.shapeblue.com 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue On Mon, 28 Dec 2020 at 06:33, Hean Seng wrote: > Just for update, I tested on Ubuntu 18, and upgrade to ACS5.15, no issue . > > After that MGMT, do-release-upgrade to Ubuntu 20, and no issue on upgrade > to Ubuntu20, However MySQL need to downgrade to MySQL 5.7. and restore back > the DB . Not sure how to make it work on MySQL8 yet. > > Create VM, Snapshot, Delivete, those is no issue as well. > > > > On Sun, Dec 27, 2020 at 1:08 AM Hean Seng wrote: > > > I think the main issue is the first time not recognize the systemvm , > > although already install they SystemVM > > > > I do bare new installation on 4.15, CentOS7,KVM, , and it works > > > > On Sun, Dec 27, 2020 at 12:31 AM Sergey Levitskiy > > wrote: > > > >> You can try this. Restore your DB backup, register SSVM template and run > >> the following against your MySQL DB before starting the upgrade. > >> > >> ALTER TABLE `cloud`.`project_account` > >> ADD CONSTRAINT `fk_project_account__account_id` FOREIGN > >> KEY(`account_id`) REFERENCES `account`(`id`) ON DELETE CASCADE , > >> ADD CONSTRAINT `uc_project_account__project_id_account_id_user_id` > >> UNIQUE (`project_id`, `account_id`, `user_id`) ; > >> > >> > >> If it still fails capture and post full management server log. > >> > >> > >> Thanks, > >> Sergey > >> > >> On 12/26/20, 2:27 AM, "Hean Seng" wrote: > >> > >> I restore the backup db, and reregister the system template using > >> cloud-install-sys-tmplt > >> , it sill getting error. > >> > >> stemVm template not found. Ovm3 hypervisor is not used, so not > failing > >> upgrade > >> > >> 2020-12-26 10:11:37,713 DEBUG [c.c.u.d.Upgrade41400to41500] > >> (main:null) > >> (logid:) Updating KVM System Vms > >> > >> 2020-12-26 10:11:37,720 ERROR [c.c.u.DatabaseUpgradeChecker] > >> (main:null) > >> (logid:) Unable to upgrade the database > >> > >> com.cloud.utils.exception.CloudRuntimeException: 4.15.0.0KVM > SystemVm > >> template not found. Cannot upgrade system Vms > >> > >> at > >> > >> > com.cloud.upgrade.dao.Upgrade41400to41500.updateSystemVmTemplates(Upgrade41400to41500.java:214) > >> > >> at > >> > >> > com.cloud.upgrade.dao.Upgrade41400to41500.performDataMigration(Upgrade41400to41500.java:70) > >> > >> On Sat, Dec 26, 2020 at 5:48 PM Hean Seng > wrote: > >> > >> > For first time I upgrade and start the MGMT server , it show > >> > following error: > >> > > >> > 2020-12-26 09:02:32,499 DEBUG [c.c.u.d.Upgrade41400to41500] > >> (main:null) > >> > (logid:) Updating System Vm template IDs > >> > > >> > 2020-12-26 09:02:32,503 DEBUG [c.c.u.d.Upgrade41400to41500] > >> (main:null) > >> > (logid:) Updating KVM System Vms > >> > > >> > 2020-12-26 09:02:32,511 ERROR [c.c.u.DatabaseUpgradeChecker] > >> (main:null) > >> > (logid:) Unable to upgrade the database > >> > > >> > com.cloud.utils.exception.CloudRuntimeException: 4.15.0.0KVM > >> SystemVm > >> > template not found. Cannot upgrade system Vms > >> > > >> > at > >> > > >> > com.cloud.upgrade.dao.Upgrade41400to41500.updateSystemVmTemplates(Upgrade41400to41500.java:214) > >> > > >> > at > >> > > >> > com.cloud.upgrade.dao.Upgrade41400to41500.performDataMigration(Upgrade41400to41500.java:70) > >> > > >> > at > >> > > >> >
Re: [VOTE] Apache Cloudstack 4.15.0.0 and UI [RC3]
MySQL 8 should work - support for it was introduced in 4.15, if not mistaken (5.7 is still a safe bet). Your issue seems to be an unclean restore of the DB, based on the input you've shared. You need to drop your cloud/cloud_usage DBs, create empty ones, restore them both from backup, start the old mgmt server, make sure that the proper systemVM template is registered with the EXACT name as specified in the upgrade guide (you have to use that name, otherwise later the DB upgrade will fail) and only then proceed with the upgrade to 4.15. On Mon, 28 Dec 2020 at 06:33, Hean Seng wrote: > Just for update, I tested on Ubuntu 18, and upgrade to ACS5.15, no issue . > > After that MGMT, do-release-upgrade to Ubuntu 20, and no issue on upgrade > to Ubuntu20, However MySQL need to downgrade to MySQL 5.7. and restore back > the DB . Not sure how to make it work on MySQL8 yet. > > Create VM, Snapshot, Delivete, those is no issue as well. > > > > On Sun, Dec 27, 2020 at 1:08 AM Hean Seng wrote: > > > I think the main issue is the first time not recognize the systemvm , > > although already install they SystemVM > > > > I do bare new installation on 4.15, CentOS7,KVM, , and it works > > > > On Sun, Dec 27, 2020 at 12:31 AM Sergey Levitskiy > > wrote: > > > >> You can try this. Restore your DB backup, register SSVM template and run > >> the following against your MySQL DB before starting the upgrade. > >> > >> ALTER TABLE `cloud`.`project_account` > >> ADD CONSTRAINT `fk_project_account__account_id` FOREIGN > >> KEY(`account_id`) REFERENCES `account`(`id`) ON DELETE CASCADE , > >> ADD CONSTRAINT `uc_project_account__project_id_account_id_user_id` > >> UNIQUE (`project_id`, `account_id`, `user_id`) ; > >> > >> > >> If it still fails capture and post full management server log. > >> > >> > >> Thanks, > >> Sergey > >> > >> On 12/26/20, 2:27 AM, "Hean Seng" wrote: > >> > >> I restore the backup db, and reregister the system template using > >> cloud-install-sys-tmplt > >> , it sill getting error. > >> > >> stemVm template not found. Ovm3 hypervisor is not used, so not > failing > >> upgrade > >> > >> 2020-12-26 10:11:37,713 DEBUG [c.c.u.d.Upgrade41400to41500] > >> (main:null) > >> (logid:) Updating KVM System Vms > >> > >> 2020-12-26 10:11:37,720 ERROR [c.c.u.DatabaseUpgradeChecker] > >> (main:null) > >> (logid:) Unable to upgrade the database > >> > >> com.cloud.utils.exception.CloudRuntimeException: 4.15.0.0KVM > SystemVm > >> template not found. Cannot upgrade system Vms > >> > >> at > >> > >> > com.cloud.upgrade.dao.Upgrade41400to41500.updateSystemVmTemplates(Upgrade41400to41500.java:214) > >> > >> at > >> > >> > com.cloud.upgrade.dao.Upgrade41400to41500.performDataMigration(Upgrade41400to41500.java:70) > >> > >> On Sat, Dec 26, 2020 at 5:48 PM Hean Seng > wrote: > >> > >> > For first time I upgrade and start the MGMT server , it show > >> > following error: > >> > > >> > 2020-12-26 09:02:32,499 DEBUG [c.c.u.d.Upgrade41400to41500] > >> (main:null) > >> > (logid:) Updating System Vm template IDs > >> > > >> > 2020-12-26 09:02:32,503 DEBUG [c.c.u.d.Upgrade41400to41500] > >> (main:null) > >> > (logid:) Updating KVM System Vms > >> > > >> > 2020-12-26 09:02:32,511 ERROR [c.c.u.DatabaseUpgradeChecker] > >> (main:null) > >> > (logid:) Unable to upgrade the database > >> > > >> > com.cloud.utils.exception.CloudRuntimeException: 4.15.0.0KVM > >> SystemVm > >> > template not found. Cannot upgrade system Vms > >> > > >> > at > >> > > >> > com.cloud.upgrade.dao.Upgrade41400to41500.updateSystemVmTemplates(Upgrade41400to41500.java:214) > >> > > >> > at > >> > > >> > com.cloud.upgrade.dao.Upgrade41400to41500.performDataMigration(Upgrade41400to41500.java:70) > >> > > >> > at > >> > > >> > com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:262) > >> > > >> > at > >> > > >> > com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:342) > >> > > >> > at > >> > > >> > org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:64) > >> > > >> > at > >> > > >> > org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:54) > >> > > >> > at > >> > > >> > org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:182) > >> > > >> > at > >> > > >> > org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:53) > >> > > >> > at > >> > > >> > org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:360) > >> > > >> > at > >> > > >> >
