Hi, Can you try to manually start the cloud service, for example: "service cloud start" and tail/share the logs which may explain why the java process is not running. If that does not work, you may also try to validate/verify the certificates (including any chain/intermediate certificates) you've uploaded and destroy the old CPVM/SSVM.
For more information on SSL certificate setup, you may read this 4.11-specific blog https://www.shapeblue.com/securing-cloudstack-4-11-with-https-tls/ which I think is applicable for 4.9 as well. Regards. ________________________________ From: Cloud List <cloud-l...@sg.or.id> Sent: Saturday, December 26, 2020 09:42 To: users@cloudstack.apache.org <users@cloudstack.apache.org>; dev <d...@cloudstack.apache.org> Subject: SSVM and CPVM agent unable to start after console proxy SSL certificate update Hi, Merry Christmas to all. We are using Cloudstack with KVM hypervisor. Since our console proxy SSL certificate has expired, we updated our new SSL certificate using below method: http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/systemvm.html#using-a-ssl-certificate-for-the-console-proxy We have done the above method in the past years without any issues, however this time round, both the SSVM and CPVM agents are not able to start after the update. The state for both VMs are up but agents are in "disconnected" state. We are still able to login to the SSVM, and found out that the cloud service is not running. root@s-4200-VM:~# service cloud status CloudStack cloud service is not running Tried to start the service: root@s-4200-VM:~# service cloud start Starting CloudStack cloud service (type=secstorage) Success But the service is not started: root@s-4200-VM:~# service cloud status CloudStack cloud service is not running Below is the logs from /var/log/cloud.log: ===== Sat Dec 26 03:45:04 UTC 2020 Executing cloud-early-config Sat Dec 26 03:45:04 UTC 2020 Detected that we are running inside kvm guest Sat Dec 26 03:45:04 UTC 2020 Found a non empty cmdline file. Will now exit the loop and proceed with configuration. Sat Dec 26 03:45:04 UTC 2020 Patching cloud service Sat Dec 26 03:45:10 UTC 2020 Updating log4j-cloud.xml Sat Dec 26 03:45:10 UTC 2020 Setting up secondary storage system vm Sat Dec 26 03:45:10 UTC 2020 checking that eth0 has IP Sat Dec 26 03:45:11 UTC 2020 waiting for eth0 interface setup with ip timer=0 Sat Dec 26 03:45:11 UTC 2020 checking that eth1 has IP Sat Dec 26 03:45:11 UTC 2020 checking that eth2 has IP Sat Dec 26 03:45:20 UTC 2020 checking that eth3 has IP Sat Dec 26 03:45:20 UTC 2020 Successfully setup storage network with STORAGE_IP:10.19.22.67, STORAGE_NETMASK:255.255.240.0, STORAGE_CIDR: Sat Dec 26 03:45:20 UTC 2020 Setting up route of RFC1918 space to 10.19.16.1 Sat Dec 26 03:45:20 UTC 2020 Setting up apache web server Sat Dec 26 03:45:20 UTC 2020 setting up apache2 for post upload of volume/template Sat Dec 26 03:45:20 UTC 2020 rewrite rules already exist in file /etc/apache2/sites-available/default-ssl Sat Dec 26 03:45:20 UTC 2020 adding cors rules to file: /etc/apache2/sites-available/default-ssl Sat Dec 26 03:45:21 UTC 2020 cloud: disable rp_filter Sat Dec 26 03:45:21 UTC 2020 disable rpfilter Sat Dec 26 03:45:21 UTC 2020 cloud: enable_fwding = 0 Sat Dec 26 03:45:21 UTC 2020 enable_fwding = 0 Sat Dec 26 03:45:21 UTC 2020 Enable service haproxy = 0 Sat Dec 26 03:45:21 UTC 2020 Processors = 1 Enable service = 0 Sat Dec 26 03:45:21 UTC 2020 Enable service dnsmasq = 0 Sat Dec 26 03:45:21 UTC 2020 Enable service cloud-passwd-srvr = 0 Sat Dec 26 03:45:21 UTC 2020 Enable service cloud = 1 ===== Result of /usr/local/cloud/systemvm/ssvm-check.sh: ===== root@s-4200-VM:/var/log# /usr/local/cloud/systemvm/ssvm-check.sh ================================================ First DNS server is 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 48 data bytes 56 bytes from 8.8.8.8: icmp_seq=0 ttl=122 time=0.531 ms 56 bytes from 8.8.8.8: icmp_seq=1 ttl=122 time=0.676 ms --- 8.8.8.8 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.531/0.604/0.676/0.073 ms Good: Can ping DNS server ================================================ Good: DNS resolves download.cloud.com ================================================ ERROR: NFS is not currently mounted Try manually mounting from inside the VM NFS server is X.X.201.1 PING X.X.201.1 (X.X.201.1): 48 data bytes 56 bytes from X.X.201.1: icmp_seq=0 ttl=255 time=0.463 ms 56 bytes from X.X.201.1: icmp_seq=1 ttl=255 time=0.482 ms --- X.X.201.1 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.463/0.473/0.482/0.000 ms Good: Can ping nfs server ================================================ Management server is 10.237.3.8. Checking connectivity. Good: Can connect to management server port 8250 ================================================ ERROR: Java process not running. Try restarting the SSVM. root@s-4200-VM:/var/log# ===== The result is OK except the NFS test, but we checked the IP address is not correct (X.X.201.1 which is the public IP address of the gateway rather than the actual NFS server IP). We tested mounting to the actual NFS server and it works fine. Have tried stopping and starting back the SSVM and the issue still persists. Anyone can help to advice how we can resolve the problem? Looking forward to your reply, thank you. -ip- <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free. www.avg.com<http://www.avg.com> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> rohit.ya...@shapeblue.com www.shapeblue.com 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue
