Re: SSVM and CPVM agent unable to start after console proxy SSL certificate update
Probably try destroying them once? Any warn or error message in mgt logs or ssvm /var/log/messages? Sent from my iPhone > On 26-Dec-2020, at 5:12 AM, Cloud List wrote: > > Hi, > > Merry Christmas to all. > > We are using Cloudstack with KVM hypervisor. Since our console proxy SSL > certificate has expired, we updated our new SSL certificate using below > method: > > http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/systemvm.html#using-a-ssl-certificate-for-the-console-proxy > > We have done the above method in the past years without any issues, however > this time round, both the SSVM and CPVM agents are not able to start after > the update. > > The state for both VMs are up but agents are in "disconnected" state. We > are still able to login to the SSVM, and found out that the cloud service > is not running. > > root@s-4200-VM:~# service cloud status > CloudStack cloud service is not running > > Tried to start the service: > > root@s-4200-VM:~# service cloud start > Starting CloudStack cloud service (type=secstorage) Success > > But the service is not started: > > root@s-4200-VM:~# service cloud status > CloudStack cloud service is not running > > Below is the logs from /var/log/cloud.log: > > = > Sat Dec 26 03:45:04 UTC 2020 Executing cloud-early-config > Sat Dec 26 03:45:04 UTC 2020 Detected that we are running inside kvm guest > Sat Dec 26 03:45:04 UTC 2020 Found a non empty cmdline file. Will now exit > the loop and proceed with configuration. > Sat Dec 26 03:45:04 UTC 2020 Patching cloud service > Sat Dec 26 03:45:10 UTC 2020 Updating log4j-cloud.xml > Sat Dec 26 03:45:10 UTC 2020 Setting up secondary storage system vm > Sat Dec 26 03:45:10 UTC 2020 checking that eth0 has IP > Sat Dec 26 03:45:11 UTC 2020 waiting for eth0 interface setup with ip > timer=0 > Sat Dec 26 03:45:11 UTC 2020 checking that eth1 has IP > Sat Dec 26 03:45:11 UTC 2020 checking that eth2 has IP > Sat Dec 26 03:45:20 UTC 2020 checking that eth3 has IP > Sat Dec 26 03:45:20 UTC 2020 Successfully setup storage network with > STORAGE_IP:10.19.22.67, STORAGE_NETMASK:255.255.240.0, STORAGE_CIDR: > Sat Dec 26 03:45:20 UTC 2020 Setting up route of RFC1918 space to 10.19.16.1 > Sat Dec 26 03:45:20 UTC 2020 Setting up apache web server > Sat Dec 26 03:45:20 UTC 2020 setting up apache2 for post upload of > volume/template > Sat Dec 26 03:45:20 UTC 2020 rewrite rules already exist in file > /etc/apache2/sites-available/default-ssl > Sat Dec 26 03:45:20 UTC 2020 adding cors rules to file: > /etc/apache2/sites-available/default-ssl > Sat Dec 26 03:45:21 UTC 2020 cloud: disable rp_filter > Sat Dec 26 03:45:21 UTC 2020 disable rpfilter > Sat Dec 26 03:45:21 UTC 2020 cloud: enable_fwding = 0 > Sat Dec 26 03:45:21 UTC 2020 enable_fwding = 0 > Sat Dec 26 03:45:21 UTC 2020 Enable service haproxy = 0 > Sat Dec 26 03:45:21 UTC 2020 Processors = 1 Enable service = 0 > Sat Dec 26 03:45:21 UTC 2020 Enable service dnsmasq = 0 > Sat Dec 26 03:45:21 UTC 2020 Enable service cloud-passwd-srvr = 0 > Sat Dec 26 03:45:21 UTC 2020 Enable service cloud = 1 > = > > Result of /usr/local/cloud/systemvm/ssvm-check.sh: > > = > root@s-4200-VM:/var/log# /usr/local/cloud/systemvm/ssvm-check.sh > > First DNS server is 8.8.8.8 > PING 8.8.8.8 (8.8.8.8): 48 data bytes > 56 bytes from 8.8.8.8: icmp_seq=0 ttl=122 time=0.531 ms > 56 bytes from 8.8.8.8: icmp_seq=1 ttl=122 time=0.676 ms > --- 8.8.8.8 ping statistics --- > 2 packets transmitted, 2 packets received, 0% packet loss > round-trip min/avg/max/stddev = 0.531/0.604/0.676/0.073 ms > Good: Can ping DNS server > > Good: DNS resolves download.cloud.com > > ERROR: NFS is not currently mounted > Try manually mounting from inside the VM > NFS server is X.X.201.1 > PING X.X.201.1 (X.X.201.1): 48 data bytes > 56 bytes from X.X.201.1: icmp_seq=0 ttl=255 time=0.463 ms > 56 bytes from X.X.201.1: icmp_seq=1 ttl=255 time=0.482 ms > --- X.X.201.1 ping statistics --- > 2 packets transmitted, 2 packets received, 0% packet loss > round-trip min/avg/max/stddev = 0.463/0.473/0.482/0.000 ms > Good: Can ping nfs server > > Management server is 10.237.3.8. Checking connectivity. > Good: Can connect to management server port 8250 > > ERROR: Java process not running. Try restarting the SSVM. > root@s-4200-VM:/var/log# > = > > The result is OK except the NFS test, but we checked the IP address is not > correct (X.X.201.1 which is the public IP address of the gateway rather > than the actual NFS server IP). We tested mounting to the actual NFS server > and it works fine. > > Have tried stopping and starting back the SSVM and the issue still persists. > > Anyone can help to advice how we can resolve the problem? > > Looking forward to y
Re: [VOTE] Apache Cloudstack 4.15.0.0 and UI [RC3]
Just for update, I tested on Ubuntu 18, and upgrade to ACS5.15, no issue . After that MGMT, do-release-upgrade to Ubuntu 20, and no issue on upgrade to Ubuntu20, However MySQL need to downgrade to MySQL 5.7. and restore back the DB . Not sure how to make it work on MySQL8 yet. Create VM, Snapshot, Delivete, those is no issue as well. On Sun, Dec 27, 2020 at 1:08 AM Hean Seng wrote: > I think the main issue is the first time not recognize the systemvm , > although already install they SystemVM > > I do bare new installation on 4.15, CentOS7,KVM, , and it works > > On Sun, Dec 27, 2020 at 12:31 AM Sergey Levitskiy > wrote: > >> You can try this. Restore your DB backup, register SSVM template and run >> the following against your MySQL DB before starting the upgrade. >> >> ALTER TABLE `cloud`.`project_account` >> ADD CONSTRAINT `fk_project_account__account_id` FOREIGN >> KEY(`account_id`) REFERENCES `account`(`id`) ON DELETE CASCADE , >> ADD CONSTRAINT `uc_project_account__project_id_account_id_user_id` >> UNIQUE (`project_id`, `account_id`, `user_id`) ; >> >> >> If it still fails capture and post full management server log. >> >> >> Thanks, >> Sergey >> >> On 12/26/20, 2:27 AM, "Hean Seng" wrote: >> >> I restore the backup db, and reregister the system template using >> cloud-install-sys-tmplt >> , it sill getting error. >> >> stemVm template not found. Ovm3 hypervisor is not used, so not failing >> upgrade >> >> 2020-12-26 10:11:37,713 DEBUG [c.c.u.d.Upgrade41400to41500] >> (main:null) >> (logid:) Updating KVM System Vms >> >> 2020-12-26 10:11:37,720 ERROR [c.c.u.DatabaseUpgradeChecker] >> (main:null) >> (logid:) Unable to upgrade the database >> >> com.cloud.utils.exception.CloudRuntimeException: 4.15.0.0KVM SystemVm >> template not found. Cannot upgrade system Vms >> >> at >> >> com.cloud.upgrade.dao.Upgrade41400to41500.updateSystemVmTemplates(Upgrade41400to41500.java:214) >> >> at >> >> com.cloud.upgrade.dao.Upgrade41400to41500.performDataMigration(Upgrade41400to41500.java:70) >> >> On Sat, Dec 26, 2020 at 5:48 PM Hean Seng wrote: >> >> > For first time I upgrade and start the MGMT server , it show >> > following error: >> > >> > 2020-12-26 09:02:32,499 DEBUG [c.c.u.d.Upgrade41400to41500] >> (main:null) >> > (logid:) Updating System Vm template IDs >> > >> > 2020-12-26 09:02:32,503 DEBUG [c.c.u.d.Upgrade41400to41500] >> (main:null) >> > (logid:) Updating KVM System Vms >> > >> > 2020-12-26 09:02:32,511 ERROR [c.c.u.DatabaseUpgradeChecker] >> (main:null) >> > (logid:) Unable to upgrade the database >> > >> > com.cloud.utils.exception.CloudRuntimeException: 4.15.0.0KVM >> SystemVm >> > template not found. Cannot upgrade system Vms >> > >> > at >> > >> com.cloud.upgrade.dao.Upgrade41400to41500.updateSystemVmTemplates(Upgrade41400to41500.java:214) >> > >> > at >> > >> com.cloud.upgrade.dao.Upgrade41400to41500.performDataMigration(Upgrade41400to41500.java:70) >> > >> > at >> > >> com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:262) >> > >> > at >> > >> com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:342) >> > >> > at >> > >> org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:64) >> > >> > at >> > >> org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:54) >> > >> > at >> > >> org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:182) >> > >> > at >> > >> org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:53) >> > >> > at >> > >> org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:360) >> > >> > at >> > >> org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:158) >> > >> > at >> > >> org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:122) >> > >> > at >> > >> org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:894) >> > >> > at >> > >> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553) >> > >> > at >> > >> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:144) >> > >> > at >> > >> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:121) >> > >> > at >> > >> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinition
