Hi Crisitian, To my understanding, you are trying to add a new zone and getting issues with the system VM having some SSH failures.
May I ask you to start everything fresh? I mean to prepare new secondary storage, seed the system VM template into new secondary storage and then deploy the zone from the beginning of the zone creation wizard. Make sure your management server has the latest packages and systemvm.iso is also as expected to the CloudStack version you are trying. Thanks, Harikrishna ________________________________ From: Cristian Ciobanu <cristian.c@istream.today> Sent: Friday, September 24, 2021 5:19 PM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: Failed to authentication SSH user root on host - repetitive error Hi Pearl, I did upgrades multiple times, I always add the template before upgrade. If was something related to this, i was expecting not to work for all zones, but right now is working for 2 zones from the same setup, environment. Regards, Cristian On Fri, Sep 24, 2021, 14:40 Pearl d'Silva <pearl.dsi...@shapeblue.com> wrote: > Hi Cristian, > > After going through your earlier mails again, and correct me if I am > wrong, but I noticed in one of your mails you mentioned that the env that's > causing issues was upgraded from a lower version, right? So, ideally when > doing upgrades, prior to initiating the upgrade we first take a backup of > the DB, register the template (new systemVM template) following this > https://docs.cloudstack.apache.org/en/latest/upgrading/upgrade/upgrade-4.15.html#update-system-vm-templates > and then upgrade the environment. However, you mentioned that you used the > cloud-install-sys-tmplt script to seed and register the template. I am not > familiar if using that script during upgrades works (it may, but I haven't > ever used that approach). > If it is possible, it would be advisable to rollback your env to the > earlier version (disable your zone, destroy the systemVMs before that) and > then follow the upgrade steps as mentioned in the documentation: > https://docs.cloudstack.apache.org/en/latest/upgrading/index.html > > Thanks, > Pearl > > > ________________________________ > From: cristian.c@istream.today <cristian.c@istream.today> > Sent: Friday, September 24, 2021 3:08 PM > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > Subject: RE: Failed to authentication SSH user root on host - repetitive > error > > Hi Pearl, > > I just did a SSVM redeploy in the same environment, but different zone and > and I see the following differences : > > "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)" > > > Working zone - https://pastebin.com/raw/88N6Fg6z > > "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB) > > datastore-6588","managed":"false","storagePort":"0","volumeSize":"(2.44 > GB) > > 021-09-24 05:22:23,959 INFO [c.c.h.v.r.VmwareResource] > (DirectAgent-437:ctx-2f1519ba fra-de-001.test.host, job-25099/job-35371, > cmd: StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the > same as the cloned VM datastore UUID: fra-de-001 - > 9d679070c7a6441592646bfbb82ef1b3 > > > > Not working zone - https://pastebin.com/raw/Um2uHgWM > > "isManaged":"false"}},"name":"ROOT-2056","size":"(0 bytes)" here is 0 > bytes... > > datastore-6694","managed":"false","storagePort":"0","volumeSize":"(0 > bytes) > here is 0 bytes... > > 2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource] > (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, > cmd: > StartCommand) (logid:d442432d) Mapped disk size is not the same as the > cloned VM disk size: 0 - 2621440000 ( here I see this line, with disk > size 0 ) > > 2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource] > (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, > cmd: > StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the same > as > the cloned VM datastore UUID: lnd-uk-002 - fb7b551a584e4b5192b8f371af56b303 > > > > I do not understand why I have here the size "0" > > > Regards, > Cristian > > > > > -----Original Message----- > From: Pearl d'Silva <pearl.dsi...@shapeblue.com> > Sent: Friday, September 24, 2021 9:44 AM > To: users@cloudstack.apache.org > Subject: Re: Failed to authentication SSH user root on host - repetitive > error > > By any chance, are there any old systemvm.isos in the secondary store, if > yes, can you please delete them. Based on the information you've provided, > it seems like, the key has been injected into systemvm.iso, but during boot > up, a script (cloud-early-config) that sets up the VM before bootstrapping > / > patching isn't copying the auth key. > Can you try the steps mentioned under the VMWare section of the doc: > https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso > and see if it helps. > SystemVm.iso - Apache Cloudstack - Apache Software > Foundation< > https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.i > so> > Mechanism of propagating systemvm.iso. Systemvm.iso gets propagated > differently for different hypervisors. It comes packaged as part of the > Cloudstack rpm or is built on dev environments with the command "mvn clean > install -P developer,systemvm" (More info on building cloudstack) .Once the > iso is at the appropriate location depending on the hypervisor it is > inserted as a cd drive and the ... > cwiki.apache.org > Thanks, > Pearl > > ________________________________ > From: cristian.c@istream.today <cristian.c@istream.today> > Sent: Friday, September 24, 2021 11:45 AM > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > Subject: RE: Failed to authentication SSH user root on host - repetitive > error > > Hi Pearl, > > I already did a check via console into the secondary storage vm, I > saw > that there are not keys in authorized_keys. > > If I try to inject, I see this : /bin/bash > /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh > /var/cloudstack/management/.ssh/id_rsa.pub > /var/cloudstack/management/.ssh/id_rsa > /usr/share/cloudstack-common/vms/systemvm.iso > mount: /dev/loop0 is write-protected, mounting read-only New public key is > the same as the one in the systemvm.iso, not injecting it, not modifying > systemvm.iso > > > [root@cloud-emea systemvm]# cd /mnt/iso/ [root@cloud-emea iso]# ls > agent.zip > authorized_keys cloud-scripts.tgz [root@cloud-emea iso]# cat > authorized_keys ssh-rsa > > AAAAB3NzaC1yc2EAAAABIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZhcPueIl > > BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+Bd1Y2U1Tx > xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+lYee > +zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIMtZ > +26AP > IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw== > cl...@cloud-emea.test.host[root@cloud-emea iso]# > > is the same key as I have in : /var/cloudstack/management/.ssh/id_rsa.pub > > (For security reason I have replaced characters from ssh pub key) > > > I'm stuck here, I do not understand the logic. > > > Regards, > Cristian > > > > > -----Original Message----- > From: Pearl d'Silva <pearl.dsi...@shapeblue.com> > Sent: Friday, September 24, 2021 8:05 AM > To: users@cloudstack.apache.org > Subject: Re: Failed to authentication SSH user root on host - repetitive > error > > Hi Cristian, > > The exception you are seeing is most likely to happen if the systemvm.iso > hasn't been injected with the Management server's key. One way to validate > it, would be to go to your secondary store - I presume you are working on a > VMware setup - mount the systemvm-4.15.2.iso and verify the authorized_keys > data with what's present on the MS at > /var/cloudstack/management/.ssh/id_rsa.pub. > > Thanks, > Pearl > > ________________________________ > From: cristian.c@istream.today <cristian.c@istream.today> > Sent: Thursday, September 23, 2021 9:50 PM > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > Subject: Failed to authentication SSH user root on host - repetitive error > > Hello, > > > > > > I was not able to fix this error "Failed to authentication SSH" and > looks like a bug for me, I will explain here why. > > > > My setup before adding new ZONE: > > > > 1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This > is > an old setup, it was upgraded frequently, I think from 4.2 ) > 2. I add new VMware zone, but this one with Advanced Networking, I end > with this repetitive error and nothing else. > 3. I decide to add a new zone, the same as I have at (2) with Advanced > Networking, everything works perfect, no issue at systemvm deploy, routers, > instances. > 4. I delete the zone from point (2) > 5. I add again the same zone, end with the same error. > > > > I have tried multiple things, delete the template, add again, inject, > nothing works, I end with the same error, over and over, and I do not see > the why. > > > > > > 021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource] > (DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, job-35231/job-35263, > cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to > Exception: > java.lang.Exception > > Message: Failed to authentication SSH user root on host 10.15.0.204 > > > > 2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb) > (logid:df4131be) Retrying after catching exception while trying to secure > agent for systemvm id=2025 > > com.cloud.utils.exception.CloudRuntimeException: Unable to read/process > CSR: > Command failed due to Exception: java.lang.Exception > > Message: Failed to authentication SSH user root on host 10.15.0.204 > > > > > > Here is the full log : https://pastebin.com/raw/fFfmquVB > > > > > > Maybe someone have a hint for me . > > > > Regards, > > Cristian > > > > > > > > > > >
