The issue, most probably, is due to different SSL provider or different names used for the certificates - I've seen this in past.
I would *strongly* suggest, removing all relevant records from the cloud.keystore table (all records related to the domain you are using - that probably means indeed ALL records from the table...) Then upload the SSL and it's intermediate/Root certificates again, i.e. from scratch. restart mgmt, and ensure SSVM/CPVM are destroyed Best, On Mon, 28 Dec 2020 at 11:43, Rohit Yadav <rohit.ya...@shapeblue.com> wrote: > Hi, > > Can you try to manually start the cloud service, for example: "service > cloud start" and tail/share the logs which may explain why the java process > is not running. > If that does not work, you may also try to validate/verify the > certificates (including any chain/intermediate certificates) you've > uploaded and destroy the old CPVM/SSVM. > > For more information on SSL certificate setup, you may read this > 4.11-specific blog > https://www.shapeblue.com/securing-cloudstack-4-11-with-https-tls/ which > I think is applicable for 4.9 as well. > > > Regards. > > ________________________________ > From: Cloud List <cloud-l...@sg.or.id> > Sent: Saturday, December 26, 2020 09:42 > To: users@cloudstack.apache.org <users@cloudstack.apache.org>; dev < > d...@cloudstack.apache.org> > Subject: SSVM and CPVM agent unable to start after console proxy SSL > certificate update > > Hi, > > Merry Christmas to all. > > We are using Cloudstack with KVM hypervisor. Since our console proxy SSL > certificate has expired, we updated our new SSL certificate using below > method: > > > http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/systemvm.html#using-a-ssl-certificate-for-the-console-proxy > > We have done the above method in the past years without any issues, however > this time round, both the SSVM and CPVM agents are not able to start after > the update. > > The state for both VMs are up but agents are in "disconnected" state. We > are still able to login to the SSVM, and found out that the cloud service > is not running. > > root@s-4200-VM:~# service cloud status > CloudStack cloud service is not running > > Tried to start the service: > > root@s-4200-VM:~# service cloud start > Starting CloudStack cloud service (type=secstorage) Success > > But the service is not started: > > root@s-4200-VM:~# service cloud status > CloudStack cloud service is not running > > Below is the logs from /var/log/cloud.log: > > ===== > Sat Dec 26 03:45:04 UTC 2020 Executing cloud-early-config > Sat Dec 26 03:45:04 UTC 2020 Detected that we are running inside kvm guest > Sat Dec 26 03:45:04 UTC 2020 Found a non empty cmdline file. Will now exit > the loop and proceed with configuration. > Sat Dec 26 03:45:04 UTC 2020 Patching cloud service > Sat Dec 26 03:45:10 UTC 2020 Updating log4j-cloud.xml > Sat Dec 26 03:45:10 UTC 2020 Setting up secondary storage system vm > Sat Dec 26 03:45:10 UTC 2020 checking that eth0 has IP > Sat Dec 26 03:45:11 UTC 2020 waiting for eth0 interface setup with ip > timer=0 > Sat Dec 26 03:45:11 UTC 2020 checking that eth1 has IP > Sat Dec 26 03:45:11 UTC 2020 checking that eth2 has IP > Sat Dec 26 03:45:20 UTC 2020 checking that eth3 has IP > Sat Dec 26 03:45:20 UTC 2020 Successfully setup storage network with > STORAGE_IP:10.19.22.67, STORAGE_NETMASK:255.255.240.0, STORAGE_CIDR: > Sat Dec 26 03:45:20 UTC 2020 Setting up route of RFC1918 space to > 10.19.16.1 > Sat Dec 26 03:45:20 UTC 2020 Setting up apache web server > Sat Dec 26 03:45:20 UTC 2020 setting up apache2 for post upload of > volume/template > Sat Dec 26 03:45:20 UTC 2020 rewrite rules already exist in file > /etc/apache2/sites-available/default-ssl > Sat Dec 26 03:45:20 UTC 2020 adding cors rules to file: > /etc/apache2/sites-available/default-ssl > Sat Dec 26 03:45:21 UTC 2020 cloud: disable rp_filter > Sat Dec 26 03:45:21 UTC 2020 disable rpfilter > Sat Dec 26 03:45:21 UTC 2020 cloud: enable_fwding = 0 > Sat Dec 26 03:45:21 UTC 2020 enable_fwding = 0 > Sat Dec 26 03:45:21 UTC 2020 Enable service haproxy = 0 > Sat Dec 26 03:45:21 UTC 2020 Processors = 1 Enable service = 0 > Sat Dec 26 03:45:21 UTC 2020 Enable service dnsmasq = 0 > Sat Dec 26 03:45:21 UTC 2020 Enable service cloud-passwd-srvr = 0 > Sat Dec 26 03:45:21 UTC 2020 Enable service cloud = 1 > ===== > > Result of /usr/local/cloud/systemvm/ssvm-check.sh: > > ===== > root@s-4200-VM:/var/log# /usr/local/cloud/systemvm/ssvm-check.sh > ================================================ > First DNS server is 8.8.8.8 > PING 8.8.8.8 (8.8.8.8): 48 data bytes > 56 bytes from 8.8.8.8: icmp_seq=0 ttl=122 time=0.531 ms > 56 bytes from 8.8.8.8: icmp_seq=1 ttl=122 time=0.676 ms > --- 8.8.8.8 ping statistics --- > 2 packets transmitted, 2 packets received, 0% packet loss > round-trip min/avg/max/stddev = 0.531/0.604/0.676/0.073 ms > Good: Can ping DNS server > ================================================ > Good: DNS resolves download.cloud.com > ================================================ > ERROR: NFS is not currently mounted > Try manually mounting from inside the VM > NFS server is X.X.201.1 > PING X.X.201.1 (X.X.201.1): 48 data bytes > 56 bytes from X.X.201.1: icmp_seq=0 ttl=255 time=0.463 ms > 56 bytes from X.X.201.1: icmp_seq=1 ttl=255 time=0.482 ms > --- X.X.201.1 ping statistics --- > 2 packets transmitted, 2 packets received, 0% packet loss > round-trip min/avg/max/stddev = 0.463/0.473/0.482/0.000 ms > Good: Can ping nfs server > ================================================ > Management server is 10.237.3.8. Checking connectivity. > Good: Can connect to management server port 8250 > ================================================ > ERROR: Java process not running. Try restarting the SSVM. > root@s-4200-VM:/var/log# > ===== > > The result is OK except the NFS test, but we checked the IP address is not > correct (X.X.201.1 which is the public IP address of the gateway rather > than the actual NFS server IP). We tested mounting to the actual NFS server > and it works fine. > > Have tried stopping and starting back the SSVM and the issue still > persists. > > Anyone can help to advice how we can resolve the problem? > > Looking forward to your reply, thank you. > > -ip- > > < > http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail > > > Virus-free. > www.avg.com<http://www.avg.com> > < > http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail > > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > > rohit.ya...@shapeblue.com > www.shapeblue.com > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > @shapeblue > > > > -- Andrija Panić
