Upgrading XenServer Clusters managed by ACS...
Hi All, I have a production deployment of ACS managing three XenServer Clusters (XenServer pools of 6 hosts each) in two different Zones. I now find myself in the position of needing to do a major version upgrade of those hosts. Happily I have a ACS lab managing a XenServer cluster running the same (old) version of XenServer that I can practice on. I have plenty of practice operating ACS to “quiesce things” for XenServer patches (start with the pool master, move guest VMs off, put that host into maintenance mode, unmanage the cluster, patch the host, then reverse & move onto the next host with the same steps except we don’t bother w/un-managing/re-managing the cluster), but as I understand a XenServer version upgrade backs up the whole original XenServer installation to another partition and makes a clean installation of XenServer on the original partition (the problem there being that when the upgraded XenServer boots up all the ACS provided/copied scripts are not there & ACS can’t manage the host). So not much of an ask here (OK maybe at the end – have I missed something obvious or doing anything foolish?), I wanted to share a bit research & lay out a set of steps that I think will work to get a pool of XenServers in a cluster upgraded and end up in a place where ACS is happy with them. Bear with me it’s a little long, 1. In XenCenter – if HA is enabled for the XenServer pool, disable it 2. Stop ACS management/usage services 3. Do MySQL database backups 4. Start ACS management/usage services 5. Start with the pool master. 6. In ACS – Migrate all guest VMs to other hosts in the cluster 7. In ACS – Prepare to put the pool master into maintenance mode (so no new guest VMs) * A caveat here related to this item I found when researching – https://www.shapeblue.com/how-to-upgrade-an-apache-cloudstack-citrix-xenserver-cluster/ i. A recommendation is made here to edit /etc/cloudstack/management/environment.properties ii. And set manage.xenserver.pool.master=false iii. And restart CloudStack management services iv. BECAUSE if one didn’t I understand CloudStack WOULD force an election for another host to become the pool master (which is “bad” as ASCs is configured to speak to the currently configured pool master) * HOWEVER THIS MAY NOT BE NECESSARY i. Found a thread titled “A Story of a failed XenServer Upgrade” here – http://mail-archives.apache.org/mod_mbox/cloudstack-users/201601.mbox/browser ii. At the end of the thread Paul Angus states that Geoff’s ShapeBlue blog article was written in the ACS 4.3 era and that ACS’ behavior “used to be that putting a host that was the pool master into maintenance would cause CloudStack to force an election for another host to become pool master - stopping you from then upgrading the active pool master first. There wasn't an 'unmanage' button at the time.” iii. The implication, (in my humble estimation) is that, today, one no longer needs to make these edits to /etc/cloudstack/management/environment.properties 1. In ACS – Put the pool master into maintenance mode (so no new guest VMs) 2. In ACS – Un-manage the cluster 3. NOW – Upgrade the XenServer pool master to the latest release * Exercise left to the reader…(I’ll share what I end up doing) 4. In XenServer – Do the following (found this nugget in a thread last November about XenServer upgrades & how to re-setup the host – thanks Richard Lawley!) * Remove this host tag: xe host-param-remove uuid=HOSTUUID param-name=tags param-key=vmops-version-com.cloud.hypervisor.xenserver.resource.XenServer650R * This makes management server set the host back up, presumably since ACS has credentials to the host in the database it can copy all the scripts back 5. In ACS – Re-manage the cluster 6. In ACS – Exit maintenance-mode for the newly upgraded host 7. In ACS – Observe that the newly upgraded host is “Enabled” and “Up” in the UI (Infrastructure --> Hosts) 8. In ACS – Testing (e.g. move an existing router/VM to the upgraded host, create new networks/VMs on the upgraded host) 9. Rinse & repeat with the remaining XenServer pool members in the ACS cluster. * WITH THIS EXCEPTION: No need to manipulate management of the cluster in ACS 10. In XenCenter – if HA is required re-enable it Now all of this completely bypasses steps that are spelled out here (which I *suspect* might now be “old”?): * http://docs.cloudstack.apache.org/en/lates
Re: cloud: Password server at 192.xx.1xx.79 did not have any password for the VM - After upgrade to ACS 4.14
I found out that there is a firewall issue and sshd config issue on VR on this ACS version (4.14) when it is configured with basic networking. By default management server is able to establish ssh connection only via local IP with VR: eth1 172.11.0.167/24, but in order to run health check si trying to connect via public IPs of the VR, this is not possible because of this : sshd config : Port 3922 #AddressFamily any ListenAddress 172.11.0.167, here i changed to 0.0.0.0 iptables : -A INPUT -i eth1 -p tcp -m tcp --dport 3922 -m state --state NEW,ESTABLISHED -j ACCEPT ( rule for eth0 is missing ) in basic network it will not work without this. I have added a rule to allow also for eth0 Regarding password issue: in VR iptables there is only this rule : -A INPUT -s 158.xx.xx.224/28 -i eth0 -p tcp -m tcp --dport 8080 -m state --state NEW -j ACCEPT, only for the first, main public IP, not for all the IPs, so i have added a rule to allow 8080 on each public IP from this router. Everything works now, till i destroy the router and i have to reconfigure again. root@r-3480-VM:~# 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 1e:00:91:00:00:33 brd ff:ff:ff:ff:ff:ff inet 158.xx.xx.226/28 brd 158.xx.xx.239 scope global eth0 valid_lft forever preferred_lft forever inet 167.xxx.xx.246/28 brd 167.xxx.xx.255 scope global eth0 valid_lft forever preferred_lft forever inet 149.xx.xxx.80/27 brd 149.xx.xxx.95 scope global eth0 valid_lft forever preferred_lft forever inet 192.xx.xxx.79/26 brd 192.xx.xxx.127 scope global eth0 valid_lft forever preferred_lft forever inet 198.xx.xxx.162/27 brd 198.xx.xxx.191 scope global eth0 valid_lft forever preferred_lft forever inet 149.xx.xxx.99/27 brd 149.xx.xxx.127 scope global eth0 valid_lft forever preferred_lft forever inet 144.xxx.xx.199/27 brd 144.xxx.xx.223 scope global eth0 valid_lft forever preferred_lft forever inet 144.xxx.xxx.177/27 brd 144.xxx.xxx.191 scope global eth0 valid_lft forever preferred_lft forever inet 66.xxx.xxx.133/27 brd 66.xx.xxx.159 scope global eth0 valid_lft forever preferred_lft forever 3: eth1: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 02:00:57:d0:02:14 brd ff:ff:ff:ff:ff:ff inet 172.11.0.167/24 brd 172.11.0.255 scope global eth1 valid_lft forever preferred_lft forever root@r-3480-VM:~# Regards, Cristian On Fri, 19 Jun 2020 at 21:40, Cristian Ciobanu wrote: > Hello, > >This is what I tried first, restart the VM before trying to reset the > pass. >The line you ask about was from messages log file. (vR) > BTW, I saw that now there is a local IP assigned to systems VM router. > Till now only public IP was assigned, not sure if this have something to do > with. > > Regards, > Cristian > > On Fri, 19 Jun 2020, 19:04 Andrija Panic, wrote: > >> After the the upgrade to 4.14, I assume you have restarted all existing >> VRs >> or networks (new VRs are created from the new systemVM template for 4.14)? >> >> If you see the password inside this file (and you do, as it seems) - that >> means that the VM (script) did not fetch the password. When the VM fetches >> the password, it will say "saved" instead of the actual password in that >> file. >> Can you just reboot once more the VM - do NOT reset the password in the >> meantime? Check logs after that and see if the password was changed. >> >> What about these 2 lines? >> >> Jun 19 14:08:04 systemvm passwd_server_ip.py: serve_password: password >> saved >> for VM IP 192.xx.xxx.80 >> >> This indicates that the password was sent to the VM, >> >> Regards, >> Andrija >> >> >> On Fri, 19 Jun 2020 at 17:11, wrote: >> >> > Hi Andrija, >> > >> >Please see : >> > >> > >> > root@r-2705-VM:~# cat /var/cache/cloud/passwords-192.xx.xxx.79 >> > 192.xx.xxx.108=Yj4AZj >> > 192.xx.xxx.101=pnj6dD >> > 192.xx.xxx.115=Q7wyGw >> > 192.xx.xxx.80=y2sS7E >> > >> > >> > y2sS7E >> > >> > VM IP : 192.xx.xxx.80 >> > >> > >> > Regards, >> > Cristian >> > >> > -Original Message- >> > From: Andrija Panic >> > Sent: Friday, June 19, 2020 5:56 PM >> > To: users >> > Subject: Re: cloud: Password server at 192.xx.1xx.79 did not have any >> > password for the VM - After upgrade to ACS 4.14 >> > >> > Can you reset a password for a VM, boot the VM and then provide the >> > content of the file /var/cache/cloud/password* from the VR? >> > >> > Regards, >> > Andrija >> > >> > On Fri, 19 Jun 2020 at 16:20, wrote: >> > >> > > Hello folks, >> > > >> > > >> > > >> > > I have successfully upgraded my Cloudstack 4.11 to 4.14 ( VMware >> > with >> > > Basic Networking ) everything works except password for VMs. I did >> > > multiple testes, differen
Re: cloud: Password server at 192.xx.1xx.79 did not have any password for the VM - After upgrade to ACS 4.14
Hello, This is what I tried first, restart the VM before trying to reset the pass. The line you ask about was from messages log file. (vR) BTW, I saw that now there is a local IP assigned to systems VM router. Till now only public IP was assigned, not sure if this have something to do with. Regards, Cristian On Fri, 19 Jun 2020, 19:04 Andrija Panic, wrote: > After the the upgrade to 4.14, I assume you have restarted all existing VRs > or networks (new VRs are created from the new systemVM template for 4.14)? > > If you see the password inside this file (and you do, as it seems) - that > means that the VM (script) did not fetch the password. When the VM fetches > the password, it will say "saved" instead of the actual password in that > file. > Can you just reboot once more the VM - do NOT reset the password in the > meantime? Check logs after that and see if the password was changed. > > What about these 2 lines? > > Jun 19 14:08:04 systemvm passwd_server_ip.py: serve_password: password > saved > for VM IP 192.xx.xxx.80 > > This indicates that the password was sent to the VM, > > Regards, > Andrija > > > On Fri, 19 Jun 2020 at 17:11, wrote: > > > Hi Andrija, > > > >Please see : > > > > > > root@r-2705-VM:~# cat /var/cache/cloud/passwords-192.xx.xxx.79 > > 192.xx.xxx.108=Yj4AZj > > 192.xx.xxx.101=pnj6dD > > 192.xx.xxx.115=Q7wyGw > > 192.xx.xxx.80=y2sS7E > > > > > > y2sS7E > > > > VM IP : 192.xx.xxx.80 > > > > > > Regards, > > Cristian > > > > -Original Message- > > From: Andrija Panic > > Sent: Friday, June 19, 2020 5:56 PM > > To: users > > Subject: Re: cloud: Password server at 192.xx.1xx.79 did not have any > > password for the VM - After upgrade to ACS 4.14 > > > > Can you reset a password for a VM, boot the VM and then provide the > > content of the file /var/cache/cloud/password* from the VR? > > > > Regards, > > Andrija > > > > On Fri, 19 Jun 2020 at 16:20, wrote: > > > > > Hello folks, > > > > > > > > > > > > I have successfully upgraded my Cloudstack 4.11 to 4.14 ( VMware > > with > > > Basic Networking ) everything works except password for VMs. I did > > > multiple testes, different OS, looks like it is not working anymore, > > > any idea why? > > > > > > > > > > > > VM log: > > > > > > > > > > > > > > > > > > Jun 19 10:57:14 localhost cloud-set-guest-password: Starting > > > cloud-set-guest-password: [ OK ] > > > > > > Jun 19 10:57:14 localhost cloud-set-guest-sshkey: Starting > > > cloud-set-guest-sshkey: [ OK ] > > > > > > Jun 19 10:57:14 localhost cloud: Sending request to ssh key server at > > > 192.xx.xxx.79 > > > > > > Jun 19 10:57:14 localhost cloud: Found password server IP > > > 192.xx.xxx.79 in > > > > > > /var/lib/NetworkManager/dhclient-6395a6b2-9b5d-4daa-86bd-343e5b823d5e- > > > eno167 > > > 77752.lease > > > > > > Jun 19 10:57:14 localhost cloud: Sending request to password server at > > > 192.xx.xxx79 > > > > > > Jun 19 10:57:15 localhost systemd: Started Dynamic System Tuning > Daemon. > > > > > > Jun 19 10:57:15 localhost systemd: Started Postfix Mail Transport > Agent. > > > > > > Jun 19 10:57:15 localhost kdumpctl: No memory reserved for crash > kernel. > > > > > > Jun 19 10:57:15 localhost kdumpctl: Starting kdump: [FAILED] > > > > > > Jun 19 10:57:15 localhost systemd: kdump.service: main process exited, > > > code=exited, status=1/FAILURE > > > > > > Jun 19 10:57:15 localhost systemd: Failed to start Crash recovery > > > kernel arming. > > > > > > Jun 19 10:57:15 localhost systemd: Unit kdump.service entered failed > > state. > > > > > > Jun 19 10:57:15 localhost systemd: kdump.service failed. > > > > > > Jun 19 10:57:42 localhost systemd: Created slice user-0.slice. > > > > > > Jun 19 10:57:42 localhost systemd: Starting user-0.slice. > > > > > > Jun 19 10:57:42 localhost systemd: Started Session 1 of user root. > > > > > > Jun 19 10:57:42 localhost systemd-logind: New session 1 of user root. > > > > > > Jun 19 10:57:42 localhost systemd: Starting Session 1 of user root. > > > > > > Jun 19 10:58:17 localhost cloud: Failed to get ssh keys from any > > > server > > > > > > Jun 19 10:58:17 localhost systemd: cloud-set-guest-sshkey.service: > > > main process exited, code=exited, status=1/FAILURE > > > > > > Jun 19 10:58:17 localhost systemd: Failed to start Cloud Set Guest > > > SSHKey Service. > > > > > > Jun 19 10:58:17 localhost systemd: Unit cloud-set-guest-sshkey.service > > > entered failed state. > > > > > > Jun 19 10:58:17 localhost systemd: cloud-set-guest-sshkey.service > failed. > > > > > > Jun 19 10:58:17 localhost cloud: Got response from server at > > > 192.xx.xxx.79 > > > > > > Jun 19 10:58:17 localhost cloud: Password server at 192.xx.xxx.79did > > > not have any password for the VM > > > > > > Jun 19 10:58:17 localhost cloud: Did not need to change password. > > > > > > Jun 19 10:58:17 localhost systemd: Started Cloud Set Guest Password > > > Service. > > > > > > Jun 19 10:58:17 localhost systemd: Reached target Multi-
Re: cloud: Password server at 192.xx.1xx.79 did not have any password for the VM - After upgrade to ACS 4.14
After the the upgrade to 4.14, I assume you have restarted all existing VRs or networks (new VRs are created from the new systemVM template for 4.14)? If you see the password inside this file (and you do, as it seems) - that means that the VM (script) did not fetch the password. When the VM fetches the password, it will say "saved" instead of the actual password in that file. Can you just reboot once more the VM - do NOT reset the password in the meantime? Check logs after that and see if the password was changed. What about these 2 lines? Jun 19 14:08:04 systemvm passwd_server_ip.py: serve_password: password saved for VM IP 192.xx.xxx.80 This indicates that the password was sent to the VM, Regards, Andrija On Fri, 19 Jun 2020 at 17:11, wrote: > Hi Andrija, > >Please see : > > > root@r-2705-VM:~# cat /var/cache/cloud/passwords-192.xx.xxx.79 > 192.xx.xxx.108=Yj4AZj > 192.xx.xxx.101=pnj6dD > 192.xx.xxx.115=Q7wyGw > 192.xx.xxx.80=y2sS7E > > > y2sS7E > > VM IP : 192.xx.xxx.80 > > > Regards, > Cristian > > -Original Message- > From: Andrija Panic > Sent: Friday, June 19, 2020 5:56 PM > To: users > Subject: Re: cloud: Password server at 192.xx.1xx.79 did not have any > password for the VM - After upgrade to ACS 4.14 > > Can you reset a password for a VM, boot the VM and then provide the > content of the file /var/cache/cloud/password* from the VR? > > Regards, > Andrija > > On Fri, 19 Jun 2020 at 16:20, wrote: > > > Hello folks, > > > > > > > > I have successfully upgraded my Cloudstack 4.11 to 4.14 ( VMware > with > > Basic Networking ) everything works except password for VMs. I did > > multiple testes, different OS, looks like it is not working anymore, > > any idea why? > > > > > > > > VM log: > > > > > > > > > > > > Jun 19 10:57:14 localhost cloud-set-guest-password: Starting > > cloud-set-guest-password: [ OK ] > > > > Jun 19 10:57:14 localhost cloud-set-guest-sshkey: Starting > > cloud-set-guest-sshkey: [ OK ] > > > > Jun 19 10:57:14 localhost cloud: Sending request to ssh key server at > > 192.xx.xxx.79 > > > > Jun 19 10:57:14 localhost cloud: Found password server IP > > 192.xx.xxx.79 in > > > > /var/lib/NetworkManager/dhclient-6395a6b2-9b5d-4daa-86bd-343e5b823d5e- > > eno167 > > 77752.lease > > > > Jun 19 10:57:14 localhost cloud: Sending request to password server at > > 192.xx.xxx79 > > > > Jun 19 10:57:15 localhost systemd: Started Dynamic System Tuning Daemon. > > > > Jun 19 10:57:15 localhost systemd: Started Postfix Mail Transport Agent. > > > > Jun 19 10:57:15 localhost kdumpctl: No memory reserved for crash kernel. > > > > Jun 19 10:57:15 localhost kdumpctl: Starting kdump: [FAILED] > > > > Jun 19 10:57:15 localhost systemd: kdump.service: main process exited, > > code=exited, status=1/FAILURE > > > > Jun 19 10:57:15 localhost systemd: Failed to start Crash recovery > > kernel arming. > > > > Jun 19 10:57:15 localhost systemd: Unit kdump.service entered failed > state. > > > > Jun 19 10:57:15 localhost systemd: kdump.service failed. > > > > Jun 19 10:57:42 localhost systemd: Created slice user-0.slice. > > > > Jun 19 10:57:42 localhost systemd: Starting user-0.slice. > > > > Jun 19 10:57:42 localhost systemd: Started Session 1 of user root. > > > > Jun 19 10:57:42 localhost systemd-logind: New session 1 of user root. > > > > Jun 19 10:57:42 localhost systemd: Starting Session 1 of user root. > > > > Jun 19 10:58:17 localhost cloud: Failed to get ssh keys from any > > server > > > > Jun 19 10:58:17 localhost systemd: cloud-set-guest-sshkey.service: > > main process exited, code=exited, status=1/FAILURE > > > > Jun 19 10:58:17 localhost systemd: Failed to start Cloud Set Guest > > SSHKey Service. > > > > Jun 19 10:58:17 localhost systemd: Unit cloud-set-guest-sshkey.service > > entered failed state. > > > > Jun 19 10:58:17 localhost systemd: cloud-set-guest-sshkey.service failed. > > > > Jun 19 10:58:17 localhost cloud: Got response from server at > > 192.xx.xxx.79 > > > > Jun 19 10:58:17 localhost cloud: Password server at 192.xx.xxx.79did > > not have any password for the VM > > > > Jun 19 10:58:17 localhost cloud: Did not need to change password. > > > > Jun 19 10:58:17 localhost systemd: Started Cloud Set Guest Password > > Service. > > > > Jun 19 10:58:17 localhost systemd: Reached target Multi-User System. > > > > Jun 19 10:58:17 localhost systemd: Starting Multi-User System. > > > > Jun 19 10:58:17 localhost systemd: Started Stop Read-Ahead Data > > Collection 10s After Completed Startup. > > > > Jun 19 10:58:17 localhost systemd: Starting Update UTMP about System > > Runlevel Changes... > > > > Jun 19 10:58:17 localhost systemd: Started Update UTMP about System > > Runlevel Changes. > > > > Jun 19 10:58:17 localhost systemd: Startup finished in 521ms (kernel) > > + 1.563s (initrd) + 1min 10.596s (userspace) = 1min 12.681s. > > > > > > > > > > > > Router Log : > > > > Jun 19 12:15:24 systemvm cloud: VR config: create file success > > > > Jun 19 12:15:24
RE: cloud: Password server at 192.xx.1xx.79 did not have any password for the VM - After upgrade to ACS 4.14
Hi Andrija, Please see : root@r-2705-VM:~# cat /var/cache/cloud/passwords-192.xx.xxx.79 192.xx.xxx.108=Yj4AZj 192.xx.xxx.101=pnj6dD 192.xx.xxx.115=Q7wyGw 192.xx.xxx.80=y2sS7E y2sS7E VM IP : 192.xx.xxx.80 Regards, Cristian -Original Message- From: Andrija Panic Sent: Friday, June 19, 2020 5:56 PM To: users Subject: Re: cloud: Password server at 192.xx.1xx.79 did not have any password for the VM - After upgrade to ACS 4.14 Can you reset a password for a VM, boot the VM and then provide the content of the file /var/cache/cloud/password* from the VR? Regards, Andrija On Fri, 19 Jun 2020 at 16:20, wrote: > Hello folks, > > > > I have successfully upgraded my Cloudstack 4.11 to 4.14 ( VMware with > Basic Networking ) everything works except password for VMs. I did > multiple testes, different OS, looks like it is not working anymore, > any idea why? > > > > VM log: > > > > > > Jun 19 10:57:14 localhost cloud-set-guest-password: Starting > cloud-set-guest-password: [ OK ] > > Jun 19 10:57:14 localhost cloud-set-guest-sshkey: Starting > cloud-set-guest-sshkey: [ OK ] > > Jun 19 10:57:14 localhost cloud: Sending request to ssh key server at > 192.xx.xxx.79 > > Jun 19 10:57:14 localhost cloud: Found password server IP > 192.xx.xxx.79 in > > /var/lib/NetworkManager/dhclient-6395a6b2-9b5d-4daa-86bd-343e5b823d5e- > eno167 > 77752.lease > > Jun 19 10:57:14 localhost cloud: Sending request to password server at > 192.xx.xxx79 > > Jun 19 10:57:15 localhost systemd: Started Dynamic System Tuning Daemon. > > Jun 19 10:57:15 localhost systemd: Started Postfix Mail Transport Agent. > > Jun 19 10:57:15 localhost kdumpctl: No memory reserved for crash kernel. > > Jun 19 10:57:15 localhost kdumpctl: Starting kdump: [FAILED] > > Jun 19 10:57:15 localhost systemd: kdump.service: main process exited, > code=exited, status=1/FAILURE > > Jun 19 10:57:15 localhost systemd: Failed to start Crash recovery > kernel arming. > > Jun 19 10:57:15 localhost systemd: Unit kdump.service entered failed state. > > Jun 19 10:57:15 localhost systemd: kdump.service failed. > > Jun 19 10:57:42 localhost systemd: Created slice user-0.slice. > > Jun 19 10:57:42 localhost systemd: Starting user-0.slice. > > Jun 19 10:57:42 localhost systemd: Started Session 1 of user root. > > Jun 19 10:57:42 localhost systemd-logind: New session 1 of user root. > > Jun 19 10:57:42 localhost systemd: Starting Session 1 of user root. > > Jun 19 10:58:17 localhost cloud: Failed to get ssh keys from any > server > > Jun 19 10:58:17 localhost systemd: cloud-set-guest-sshkey.service: > main process exited, code=exited, status=1/FAILURE > > Jun 19 10:58:17 localhost systemd: Failed to start Cloud Set Guest > SSHKey Service. > > Jun 19 10:58:17 localhost systemd: Unit cloud-set-guest-sshkey.service > entered failed state. > > Jun 19 10:58:17 localhost systemd: cloud-set-guest-sshkey.service failed. > > Jun 19 10:58:17 localhost cloud: Got response from server at > 192.xx.xxx.79 > > Jun 19 10:58:17 localhost cloud: Password server at 192.xx.xxx.79did > not have any password for the VM > > Jun 19 10:58:17 localhost cloud: Did not need to change password. > > Jun 19 10:58:17 localhost systemd: Started Cloud Set Guest Password > Service. > > Jun 19 10:58:17 localhost systemd: Reached target Multi-User System. > > Jun 19 10:58:17 localhost systemd: Starting Multi-User System. > > Jun 19 10:58:17 localhost systemd: Started Stop Read-Ahead Data > Collection 10s After Completed Startup. > > Jun 19 10:58:17 localhost systemd: Starting Update UTMP about System > Runlevel Changes... > > Jun 19 10:58:17 localhost systemd: Started Update UTMP about System > Runlevel Changes. > > Jun 19 10:58:17 localhost systemd: Startup finished in 521ms (kernel) > + 1.563s (initrd) + 1min 10.596s (userspace) = 1min 12.681s. > > > > > > Router Log : > > Jun 19 12:15:24 systemvm cloud: VR config: create file success > > Jun 19 12:15:24 systemvm cloud: VR config: executing: > /opt/cloud/bin/update_config.py > vm_metadata.json.a997727c-51e2-4730-b1e4-5a033cf8672f > > Jun 19 12:15:24 systemvm cloud: VR config: execution success > > Jun 19 12:15:24 systemvm cloud: VR config: creating file: > /var/cache/cloud/vm_metadata.json.079f12c9-45d1-48e0-986c-0ed68f764128 > > Jun 19 12:15:24 systemvm cloud: VR config: create file success > > Jun 19 12:15:24 systemvm cloud: VR config: executing: > /opt/cloud/bin/update_config.py > vm_metadata.json.079f12c9-45d1-48e0-986c-0ed68f764128 > > Jun 19 12:08:48 systemvm cloud: VR config: execution success > > Jun 19 12:08:50 systemvm cloud: VR config: Flushing conntrack table > > Jun 19 12:08:50 systemvm cloud: VR config: Flushing conntrack table > completed > > Jun 19 12:13:46 systemvm kernel: [ 320.771392] nf_conntrack: default > automatic helper assignment has been turned off for security reasons > and CT-based firewall rule not found. Use the iptables CT target to > attach helpers instead. > > Jun 19 13:38:36 sys
Re: cloud: Password server at 192.xx.1xx.79 did not have any password for the VM - After upgrade to ACS 4.14
Can you reset a password for a VM, boot the VM and then provide the content of the file /var/cache/cloud/password* from the VR? Regards, Andrija On Fri, 19 Jun 2020 at 16:20, wrote: > Hello folks, > > > > I have successfully upgraded my Cloudstack 4.11 to 4.14 ( VMware with > Basic Networking ) everything works except password for VMs. I did > multiple testes, different OS, looks like it is not working anymore, any > idea why? > > > > VM log: > > > > > > Jun 19 10:57:14 localhost cloud-set-guest-password: Starting > cloud-set-guest-password: [ OK ] > > Jun 19 10:57:14 localhost cloud-set-guest-sshkey: Starting > cloud-set-guest-sshkey: [ OK ] > > Jun 19 10:57:14 localhost cloud: Sending request to ssh key server at > 192.xx.xxx.79 > > Jun 19 10:57:14 localhost cloud: Found password server IP 192.xx.xxx.79 in > > /var/lib/NetworkManager/dhclient-6395a6b2-9b5d-4daa-86bd-343e5b823d5e-eno167 > 77752.lease > > Jun 19 10:57:14 localhost cloud: Sending request to password server at > 192.xx.xxx79 > > Jun 19 10:57:15 localhost systemd: Started Dynamic System Tuning Daemon. > > Jun 19 10:57:15 localhost systemd: Started Postfix Mail Transport Agent. > > Jun 19 10:57:15 localhost kdumpctl: No memory reserved for crash kernel. > > Jun 19 10:57:15 localhost kdumpctl: Starting kdump: [FAILED] > > Jun 19 10:57:15 localhost systemd: kdump.service: main process exited, > code=exited, status=1/FAILURE > > Jun 19 10:57:15 localhost systemd: Failed to start Crash recovery kernel > arming. > > Jun 19 10:57:15 localhost systemd: Unit kdump.service entered failed state. > > Jun 19 10:57:15 localhost systemd: kdump.service failed. > > Jun 19 10:57:42 localhost systemd: Created slice user-0.slice. > > Jun 19 10:57:42 localhost systemd: Starting user-0.slice. > > Jun 19 10:57:42 localhost systemd: Started Session 1 of user root. > > Jun 19 10:57:42 localhost systemd-logind: New session 1 of user root. > > Jun 19 10:57:42 localhost systemd: Starting Session 1 of user root. > > Jun 19 10:58:17 localhost cloud: Failed to get ssh keys from any server > > Jun 19 10:58:17 localhost systemd: cloud-set-guest-sshkey.service: main > process exited, code=exited, status=1/FAILURE > > Jun 19 10:58:17 localhost systemd: Failed to start Cloud Set Guest SSHKey > Service. > > Jun 19 10:58:17 localhost systemd: Unit cloud-set-guest-sshkey.service > entered failed state. > > Jun 19 10:58:17 localhost systemd: cloud-set-guest-sshkey.service failed. > > Jun 19 10:58:17 localhost cloud: Got response from server at 192.xx.xxx.79 > > Jun 19 10:58:17 localhost cloud: Password server at 192.xx.xxx.79did not > have any password for the VM > > Jun 19 10:58:17 localhost cloud: Did not need to change password. > > Jun 19 10:58:17 localhost systemd: Started Cloud Set Guest Password > Service. > > Jun 19 10:58:17 localhost systemd: Reached target Multi-User System. > > Jun 19 10:58:17 localhost systemd: Starting Multi-User System. > > Jun 19 10:58:17 localhost systemd: Started Stop Read-Ahead Data Collection > 10s After Completed Startup. > > Jun 19 10:58:17 localhost systemd: Starting Update UTMP about System > Runlevel Changes... > > Jun 19 10:58:17 localhost systemd: Started Update UTMP about System > Runlevel > Changes. > > Jun 19 10:58:17 localhost systemd: Startup finished in 521ms (kernel) + > 1.563s (initrd) + 1min 10.596s (userspace) = 1min 12.681s. > > > > > > Router Log : > > Jun 19 12:15:24 systemvm cloud: VR config: create file success > > Jun 19 12:15:24 systemvm cloud: VR config: executing: > /opt/cloud/bin/update_config.py > vm_metadata.json.a997727c-51e2-4730-b1e4-5a033cf8672f > > Jun 19 12:15:24 systemvm cloud: VR config: execution success > > Jun 19 12:15:24 systemvm cloud: VR config: creating file: > /var/cache/cloud/vm_metadata.json.079f12c9-45d1-48e0-986c-0ed68f764128 > > Jun 19 12:15:24 systemvm cloud: VR config: create file success > > Jun 19 12:15:24 systemvm cloud: VR config: executing: > /opt/cloud/bin/update_config.py > vm_metadata.json.079f12c9-45d1-48e0-986c-0ed68f764128 > > Jun 19 12:08:48 systemvm cloud: VR config: execution success > > Jun 19 12:08:50 systemvm cloud: VR config: Flushing conntrack table > > Jun 19 12:08:50 systemvm cloud: VR config: Flushing conntrack table > completed > > Jun 19 12:13:46 systemvm kernel: [ 320.771392] nf_conntrack: default > automatic helper assignment has been turned off for security reasons and > CT-based firewall rule not found. Use the iptables CT target to attach > helpers instead. > > Jun 19 13:38:36 systemvm passwd_server_ip.py: serve_password: password > saved > for VM IP 192.xx.xxx.101 > > Jun 19 13:47:24 systemvm passwd_server_ip.py: serve_password: password > saved > for VM IP 192.xx.xxx.101 > > Jun 19 13:53:00 systemvm passwd_server_ip.py: serve_password: password > saved > for VM IP 192.xx.xxx.108 > > Jun 19 14:05:22 systemvm passwd_server_ip.py: serve_password: password > saved > for VM IP 192.xx.xxx.108 > > Jun 19 14:08:04 systemvm passwd_server_ip.py: serve_pass
cloud: Password server at 192.xx.1xx.79 did not have any password for the VM - After upgrade to ACS 4.14
Hello folks, I have successfully upgraded my Cloudstack 4.11 to 4.14 ( VMware with Basic Networking ) everything works except password for VMs. I did multiple testes, different OS, looks like it is not working anymore, any idea why? VM log: Jun 19 10:57:14 localhost cloud-set-guest-password: Starting cloud-set-guest-password: [ OK ] Jun 19 10:57:14 localhost cloud-set-guest-sshkey: Starting cloud-set-guest-sshkey: [ OK ] Jun 19 10:57:14 localhost cloud: Sending request to ssh key server at 192.xx.xxx.79 Jun 19 10:57:14 localhost cloud: Found password server IP 192.xx.xxx.79 in /var/lib/NetworkManager/dhclient-6395a6b2-9b5d-4daa-86bd-343e5b823d5e-eno167 77752.lease Jun 19 10:57:14 localhost cloud: Sending request to password server at 192.xx.xxx79 Jun 19 10:57:15 localhost systemd: Started Dynamic System Tuning Daemon. Jun 19 10:57:15 localhost systemd: Started Postfix Mail Transport Agent. Jun 19 10:57:15 localhost kdumpctl: No memory reserved for crash kernel. Jun 19 10:57:15 localhost kdumpctl: Starting kdump: [FAILED] Jun 19 10:57:15 localhost systemd: kdump.service: main process exited, code=exited, status=1/FAILURE Jun 19 10:57:15 localhost systemd: Failed to start Crash recovery kernel arming. Jun 19 10:57:15 localhost systemd: Unit kdump.service entered failed state. Jun 19 10:57:15 localhost systemd: kdump.service failed. Jun 19 10:57:42 localhost systemd: Created slice user-0.slice. Jun 19 10:57:42 localhost systemd: Starting user-0.slice. Jun 19 10:57:42 localhost systemd: Started Session 1 of user root. Jun 19 10:57:42 localhost systemd-logind: New session 1 of user root. Jun 19 10:57:42 localhost systemd: Starting Session 1 of user root. Jun 19 10:58:17 localhost cloud: Failed to get ssh keys from any server Jun 19 10:58:17 localhost systemd: cloud-set-guest-sshkey.service: main process exited, code=exited, status=1/FAILURE Jun 19 10:58:17 localhost systemd: Failed to start Cloud Set Guest SSHKey Service. Jun 19 10:58:17 localhost systemd: Unit cloud-set-guest-sshkey.service entered failed state. Jun 19 10:58:17 localhost systemd: cloud-set-guest-sshkey.service failed. Jun 19 10:58:17 localhost cloud: Got response from server at 192.xx.xxx.79 Jun 19 10:58:17 localhost cloud: Password server at 192.xx.xxx.79did not have any password for the VM Jun 19 10:58:17 localhost cloud: Did not need to change password. Jun 19 10:58:17 localhost systemd: Started Cloud Set Guest Password Service. Jun 19 10:58:17 localhost systemd: Reached target Multi-User System. Jun 19 10:58:17 localhost systemd: Starting Multi-User System. Jun 19 10:58:17 localhost systemd: Started Stop Read-Ahead Data Collection 10s After Completed Startup. Jun 19 10:58:17 localhost systemd: Starting Update UTMP about System Runlevel Changes... Jun 19 10:58:17 localhost systemd: Started Update UTMP about System Runlevel Changes. Jun 19 10:58:17 localhost systemd: Startup finished in 521ms (kernel) + 1.563s (initrd) + 1min 10.596s (userspace) = 1min 12.681s. Router Log : Jun 19 12:15:24 systemvm cloud: VR config: create file success Jun 19 12:15:24 systemvm cloud: VR config: executing: /opt/cloud/bin/update_config.py vm_metadata.json.a997727c-51e2-4730-b1e4-5a033cf8672f Jun 19 12:15:24 systemvm cloud: VR config: execution success Jun 19 12:15:24 systemvm cloud: VR config: creating file: /var/cache/cloud/vm_metadata.json.079f12c9-45d1-48e0-986c-0ed68f764128 Jun 19 12:15:24 systemvm cloud: VR config: create file success Jun 19 12:15:24 systemvm cloud: VR config: executing: /opt/cloud/bin/update_config.py vm_metadata.json.079f12c9-45d1-48e0-986c-0ed68f764128 Jun 19 12:08:48 systemvm cloud: VR config: execution success Jun 19 12:08:50 systemvm cloud: VR config: Flushing conntrack table Jun 19 12:08:50 systemvm cloud: VR config: Flushing conntrack table completed Jun 19 12:13:46 systemvm kernel: [ 320.771392] nf_conntrack: default automatic helper assignment has been turned off for security reasons and CT-based firewall rule not found. Use the iptables CT target to attach helpers instead. Jun 19 13:38:36 systemvm passwd_server_ip.py: serve_password: password saved for VM IP 192.xx.xxx.101 Jun 19 13:47:24 systemvm passwd_server_ip.py: serve_password: password saved for VM IP 192.xx.xxx.101 Jun 19 13:53:00 systemvm passwd_server_ip.py: serve_password: password saved for VM IP 192.xx.xxx.108 Jun 19 14:05:22 systemvm passwd_server_ip.py: serve_password: password saved for VM IP 192.xx.xxx.108 Jun 19 14:08:04 systemvm passwd_server_ip.py: serve_password: password saved for VM IP 192.xx.xxx.80 2020-06-19 14:08:19,737 INFO Executing: systemctl start cloud-password-ser...@192.xx.xxx.79 2020-06-19 14:08:19,742 INFO Service cloud-password-ser...@192.xx.xxx.79 start 2020-06-19 14:08:19,742 INFO Checking if default ipv4 route is present 2020-06-19 14:08:19,742 INFO Executing: ip -4 route list 0/0 2020-06-19 14:08:19,744 INFO Default route found:
