Re: Failed to authentication SSH user root on host - repetitive error
Hi Cristian, The exception you are seeing is most likely to happen if the systemvm.iso hasn't been injected with the Management server's key. One way to validate it, would be to go to your secondary store - I presume you are working on a VMware setup - mount the systemvm-4.15.2.iso and verify the authorized_keys data with what's present on the MS at /var/cloudstack/management/.ssh/id_rsa.pub. Thanks, Pearl From: cristian.c@istream.today Sent: Thursday, September 23, 2021 9:50 PM To: users@cloudstack.apache.org Subject: Failed to authentication SSH user root on host - repetitive error Hello, I was not able to fix this error "Failed to authentication SSH" and looks like a bug for me, I will explain here why. My setup before adding new ZONE: 1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This is an old setup, it was upgraded frequently, I think from 4.2 ) 2. I add new VMware zone, but this one with Advanced Networking, I end with this repetitive error and nothing else. 3. I decide to add a new zone, the same as I have at (2) with Advanced Networking, everything works perfect, no issue at systemvm deploy, routers, instances. 4. I delete the zone from point (2) 5. I add again the same zone, end with the same error. I have tried multiple things, delete the template, add again, inject, nothing works, I end with the same error, over and over, and I do not see the why. 021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource] (DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, job-35231/job-35263, cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to Exception: java.lang.Exception Message: Failed to authentication SSH user root on host 10.15.0.204 2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb) (logid:df4131be) Retrying after catching exception while trying to secure agent for systemvm id=2025 com.cloud.utils.exception.CloudRuntimeException: Unable to read/process CSR: Command failed due to Exception: java.lang.Exception Message: Failed to authentication SSH user root on host 10.15.0.204 Here is the full log : https://pastebin.com/raw/fFfmquVB Maybe someone have a hint for me . Regards, Cristian
RE: Failed to authentication SSH user root on host - repetitive error
Hi Pearl, I already did a check via console into the secondary storage vm, I saw that there are not keys in authorized_keys. If I try to inject, I see this : /bin/bash /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh /var/cloudstack/management/.ssh/id_rsa.pub /var/cloudstack/management/.ssh/id_rsa /usr/share/cloudstack-common/vms/systemvm.iso mount: /dev/loop0 is write-protected, mounting read-only New public key is the same as the one in the systemvm.iso, not injecting it, not modifying systemvm.iso [root@cloud-emea systemvm]# cd /mnt/iso/ [root@cloud-emea iso]# ls agent.zip authorized_keys cloud-scripts.tgz [root@cloud-emea iso]# cat authorized_keys ssh-rsa B3NzaC1yc2EBIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZhcPueIl BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+x xx+Bd1Y2U1TxlYee +zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIMtZ26AP IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw== cl...@cloud-emea.test.host[root@cloud-emea iso]# is the same key as I have in : /var/cloudstack/management/.ssh/id_rsa.pub (For security reason I have replaced characters from ssh pub key) I'm stuck here, I do not understand the logic. Regards, Cristian -Original Message- From: Pearl d'Silva Sent: Friday, September 24, 2021 8:05 AM To: users@cloudstack.apache.org Subject: Re: Failed to authentication SSH user root on host - repetitive error Hi Cristian, The exception you are seeing is most likely to happen if the systemvm.iso hasn't been injected with the Management server's key. One way to validate it, would be to go to your secondary store - I presume you are working on a VMware setup - mount the systemvm-4.15.2.iso and verify the authorized_keys data with what's present on the MS at /var/cloudstack/management/.ssh/id_rsa.pub. Thanks, Pearl From: cristian.c@istream.today Sent: Thursday, September 23, 2021 9:50 PM To: users@cloudstack.apache.org Subject: Failed to authentication SSH user root on host - repetitive error Hello, I was not able to fix this error "Failed to authentication SSH" and looks like a bug for me, I will explain here why. My setup before adding new ZONE: 1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This is an old setup, it was upgraded frequently, I think from 4.2 ) 2. I add new VMware zone, but this one with Advanced Networking, I end with this repetitive error and nothing else. 3. I decide to add a new zone, the same as I have at (2) with Advanced Networking, everything works perfect, no issue at systemvm deploy, routers, instances. 4. I delete the zone from point (2) 5. I add again the same zone, end with the same error. I have tried multiple things, delete the template, add again, inject, nothing works, I end with the same error, over and over, and I do not see the why. 021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource] (DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, job-35231/job-35263, cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to Exception: java.lang.Exception Message: Failed to authentication SSH user root on host 10.15.0.204 2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb) (logid:df4131be) Retrying after catching exception while trying to secure agent for systemvm id=2025 com.cloud.utils.exception.CloudRuntimeException: Unable to read/process CSR: Command failed due to Exception: java.lang.Exception Message: Failed to authentication SSH user root on host 10.15.0.204 Here is the full log : https://pastebin.com/raw/fFfmquVB Maybe someone have a hint for me . Regards, Cristian
Re: Failed to authentication SSH user root on host - repetitive error
Hi Cristian, When the new public key has been injected into systemvm.iso, please stop/start or destroy SSVM. -Wei On Fri, 24 Sept 2021 at 08:16, wrote: > Hi Pearl, > > I already did a check via console into the secondary storage vm, I > saw > that there are not keys in authorized_keys. > > If I try to inject, I see this : /bin/bash > /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh > /var/cloudstack/management/.ssh/id_rsa.pub > /var/cloudstack/management/.ssh/id_rsa > /usr/share/cloudstack-common/vms/systemvm.iso > mount: /dev/loop0 is write-protected, mounting read-only > New public key is the same as the one in the systemvm.iso, not injecting > it, > not modifying systemvm.iso > > > [root@cloud-emea systemvm]# cd /mnt/iso/ > [root@cloud-emea iso]# ls > agent.zip authorized_keys cloud-scripts.tgz > [root@cloud-emea iso]# cat authorized_keys > ssh-rsa > > B3NzaC1yc2EBIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZhcPueIl > > BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+x > > xx+Bd1Y2U1TxlYee > > +zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIMtZ26AP > IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw== > cl...@cloud-emea.test.host[root@cloud-emea iso]# > > is the same key as I have in : /var/cloudstack/management/.ssh/id_rsa.pub > > (For security reason I have replaced characters from ssh pub key) > > > I'm stuck here, I do not understand the logic. > > > Regards, > Cristian > > -----Original Message- > From: Pearl d'Silva > Sent: Friday, September 24, 2021 8:05 AM > To: users@cloudstack.apache.org > Subject: Re: Failed to authentication SSH user root on host - repetitive > error > > Hi Cristian, > > The exception you are seeing is most likely to happen if the systemvm.iso > hasn't been injected with the Management server's key. One way to validate > it, would be to go to your secondary store - I presume you are working on a > VMware setup - mount the systemvm-4.15.2.iso and verify the authorized_keys > data with what's present on the MS at > /var/cloudstack/management/.ssh/id_rsa.pub. > > Thanks, > Pearl > > > From: cristian.c@istream.today > Sent: Thursday, September 23, 2021 9:50 PM > To: users@cloudstack.apache.org > Subject: Failed to authentication SSH user root on host - repetitive error > > Hello, > > > > > > I was not able to fix this error "Failed to authentication SSH" and > looks like a bug for me, I will explain here why. > > > > My setup before adding new ZONE: > > > > 1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This > is > an old setup, it was upgraded frequently, I think from 4.2 ) > 2. I add new VMware zone, but this one with Advanced Networking, I end > with this repetitive error and nothing else. > 3. I decide to add a new zone, the same as I have at (2) with Advanced > Networking, everything works perfect, no issue at systemvm deploy, routers, > instances. > 4. I delete the zone from point (2) > 5. I add again the same zone, end with the same error. > > > >I have tried multiple things, delete the template, add again, inject, > nothing works, I end with the same error, over and over, and I do not see > the why. > > > > > > 021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource] > (DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, job-35231/job-35263, > cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to > Exception: > java.lang.Exception > > Message: Failed to authentication SSH user root on host 10.15.0.204 > > > > 2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb) > (logid:df4131be) Retrying after catching exception while trying to secure > agent for systemvm id=2025 > > com.cloud.utils.exception.CloudRuntimeException: Unable to read/process > CSR: > Command failed due to Exception: java.lang.Exception > > Message: Failed to authentication SSH user root on host 10.15.0.204 > > > > > > Here is the full log : https://pastebin.com/raw/fFfmquVB > > > > > > Maybe someone have a hint for me . > > > > Regards, > > Cristian > > > > > > > > > >
Re: Failed to authentication SSH user root on host - repetitive error
By any chance, are there any old systemvm.isos in the secondary store, if yes, can you please delete them. Based on the information you've provided, it seems like, the key has been injected into systemvm.iso, but during boot up, a script (cloud-early-config) that sets up the VM before bootstrapping / patching isn't copying the auth key. Can you try the steps mentioned under the VMWare section of the doc: https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso and see if it helps. SystemVm.iso - Apache Cloudstack - Apache Software Foundation<https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso> Mechanism of propagating systemvm.iso. Systemvm.iso gets propagated differently for different hypervisors. It comes packaged as part of the Cloudstack rpm or is built on dev environments with the command "mvn clean install -P developer,systemvm" (More info on building cloudstack) .Once the iso is at the appropriate location depending on the hypervisor it is inserted as a cd drive and the ... cwiki.apache.org Thanks, Pearl From: cristian.c@istream.today Sent: Friday, September 24, 2021 11:45 AM To: users@cloudstack.apache.org Subject: RE: Failed to authentication SSH user root on host - repetitive error Hi Pearl, I already did a check via console into the secondary storage vm, I saw that there are not keys in authorized_keys. If I try to inject, I see this : /bin/bash /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh /var/cloudstack/management/.ssh/id_rsa.pub /var/cloudstack/management/.ssh/id_rsa /usr/share/cloudstack-common/vms/systemvm.iso mount: /dev/loop0 is write-protected, mounting read-only New public key is the same as the one in the systemvm.iso, not injecting it, not modifying systemvm.iso [root@cloud-emea systemvm]# cd /mnt/iso/ [root@cloud-emea iso]# ls agent.zip authorized_keys cloud-scripts.tgz [root@cloud-emea iso]# cat authorized_keys ssh-rsa B3NzaC1yc2EBIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZhcPueIl BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+x xx+Bd1Y2U1TxlYee +zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIMtZ26AP IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw== cl...@cloud-emea.test.host[root@cloud-emea iso]# is the same key as I have in : /var/cloudstack/management/.ssh/id_rsa.pub (For security reason I have replaced characters from ssh pub key) I'm stuck here, I do not understand the logic. Regards, Cristian -Original Message- From: Pearl d'Silva Sent: Friday, September 24, 2021 8:05 AM To: users@cloudstack.apache.org Subject: Re: Failed to authentication SSH user root on host - repetitive error Hi Cristian, The exception you are seeing is most likely to happen if the systemvm.iso hasn't been injected with the Management server's key. One way to validate it, would be to go to your secondary store - I presume you are working on a VMware setup - mount the systemvm-4.15.2.iso and verify the authorized_keys data with what's present on the MS at /var/cloudstack/management/.ssh/id_rsa.pub. Thanks, Pearl From: cristian.c@istream.today Sent: Thursday, September 23, 2021 9:50 PM To: users@cloudstack.apache.org Subject: Failed to authentication SSH user root on host - repetitive error Hello, I was not able to fix this error "Failed to authentication SSH" and looks like a bug for me, I will explain here why. My setup before adding new ZONE: 1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This is an old setup, it was upgraded frequently, I think from 4.2 ) 2. I add new VMware zone, but this one with Advanced Networking, I end with this repetitive error and nothing else. 3. I decide to add a new zone, the same as I have at (2) with Advanced Networking, everything works perfect, no issue at systemvm deploy, routers, instances. 4. I delete the zone from point (2) 5. I add again the same zone, end with the same error. I have tried multiple things, delete the template, add again, inject, nothing works, I end with the same error, over and over, and I do not see the why. 021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource] (DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, job-35231/job-35263, cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to Exception: java.lang.Exception Message: Failed to authentication SSH user root on host 10.15.0.204 2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb) (logid:df4131be) Retrying after catching exception while trying to secure agent for systemvm id=2025 com.cloud.utils.exception.CloudRuntimeException: Unable to read/process CSR:
RE: Failed to authentication SSH user root on host - repetitive error
Hi Pearl, I did the following multiple times: 1. remove systemvm.iso from secondary storage 2. remove vmdk template from ESXI host and from database spool ref. 3. I have deleted the old systemvm.iso.bak 4 restarted the MS I didn't tried to change the minor version, how can I do this? In DB? Is there a problem with this template? /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmp lt -m /var/cloudstack/mnt/345049356158.7cfbc14b -u http://packages.shapeblue.com.s3-eu-west-1.amazonaws.com/systemvmtemplate/4. 15/systemvmtemplate-4.15.1-vmware.ova -h vmware -F Thank you, Cristian -Original Message- From: Pearl d'Silva Sent: Friday, September 24, 2021 9:44 AM To: users@cloudstack.apache.org Subject: Re: Failed to authentication SSH user root on host - repetitive error By any chance, are there any old systemvm.isos in the secondary store, if yes, can you please delete them. Based on the information you've provided, it seems like, the key has been injected into systemvm.iso, but during boot up, a script (cloud-early-config) that sets up the VM before bootstrapping / patching isn't copying the auth key. Can you try the steps mentioned under the VMWare section of the doc: https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso and see if it helps. SystemVm.iso - Apache Cloudstack - Apache Software Foundation<https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.i so> Mechanism of propagating systemvm.iso. Systemvm.iso gets propagated differently for different hypervisors. It comes packaged as part of the Cloudstack rpm or is built on dev environments with the command "mvn clean install -P developer,systemvm" (More info on building cloudstack) .Once the iso is at the appropriate location depending on the hypervisor it is inserted as a cd drive and the ... cwiki.apache.org Thanks, Pearl From: cristian.c@istream.today Sent: Friday, September 24, 2021 11:45 AM To: users@cloudstack.apache.org Subject: RE: Failed to authentication SSH user root on host - repetitive error Hi Pearl, I already did a check via console into the secondary storage vm, I saw that there are not keys in authorized_keys. If I try to inject, I see this : /bin/bash /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh /var/cloudstack/management/.ssh/id_rsa.pub /var/cloudstack/management/.ssh/id_rsa /usr/share/cloudstack-common/vms/systemvm.iso mount: /dev/loop0 is write-protected, mounting read-only New public key is the same as the one in the systemvm.iso, not injecting it, not modifying systemvm.iso [root@cloud-emea systemvm]# cd /mnt/iso/ [root@cloud-emea iso]# ls agent.zip authorized_keys cloud-scripts.tgz [root@cloud-emea iso]# cat authorized_keys ssh-rsa B3NzaC1yc2EBIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZhcPueIl BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+x xx+Bd1Y2U1Tx xx+lYee +zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIMtZ +26AP IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw== cl...@cloud-emea.test.host[root@cloud-emea iso]# is the same key as I have in : /var/cloudstack/management/.ssh/id_rsa.pub (For security reason I have replaced characters from ssh pub key) I'm stuck here, I do not understand the logic. Regards, Cristian -Original Message- From: Pearl d'Silva Sent: Friday, September 24, 2021 8:05 AM To: users@cloudstack.apache.org Subject: Re: Failed to authentication SSH user root on host - repetitive error Hi Cristian, The exception you are seeing is most likely to happen if the systemvm.iso hasn't been injected with the Management server's key. One way to validate it, would be to go to your secondary store - I presume you are working on a VMware setup - mount the systemvm-4.15.2.iso and verify the authorized_keys data with what's present on the MS at /var/cloudstack/management/.ssh/id_rsa.pub. Thanks, Pearl From: cristian.c@istream.today Sent: Thursday, September 23, 2021 9:50 PM To: users@cloudstack.apache.org Subject: Failed to authentication SSH user root on host - repetitive error Hello, I was not able to fix this error "Failed to authentication SSH" and looks like a bug for me, I will explain here why. My setup before adding new ZONE: 1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This is an old setup, it was upgraded frequently, I think from 4.2 ) 2. I add new VMware zone, but this one with Advanced Networking, I end with this repetitive error and nothing else. 3. I decide to add a new zone, the same as I have at (2) with Advanced Networking, everything works perfect, no issue at sy
RE: Failed to authentication SSH user root on host - repetitive error
Hi Wei, I think here is the problem, when the key are injected, the SSVM is not running yet, is starting few seconds later So, the answer is that I do not have what to STOP, I can only delete the SSVM instance from VMware side.. Thank you, Cristian -Original Message- From: Wei ZHOU Sent: Friday, September 24, 2021 9:43 AM To: users Subject: Re: Failed to authentication SSH user root on host - repetitive error Hi Cristian, When the new public key has been injected into systemvm.iso, please stop/start or destroy SSVM. -Wei On Fri, 24 Sept 2021 at 08:16, wrote: > Hi Pearl, > > I already did a check via console into the secondary storage vm, > I saw that there are not keys in authorized_keys. > > If I try to inject, I see this : /bin/bash > /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh > /var/cloudstack/management/.ssh/id_rsa.pub > /var/cloudstack/management/.ssh/id_rsa > /usr/share/cloudstack-common/vms/systemvm.iso > mount: /dev/loop0 is write-protected, mounting read-only New public > key is the same as the one in the systemvm.iso, not injecting it, not > modifying systemvm.iso > > > [root@cloud-emea systemvm]# cd /mnt/iso/ [root@cloud-emea iso]# ls > agent.zip authorized_keys cloud-scripts.tgz [root@cloud-emea iso]# > cat authorized_keys ssh-rsa > > B3NzaC1yc2EBIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZh > cPueIl > > BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+xxx > xx > > xx+Bd1Y2U1 > xx+TxlYee > > +zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIM > +tZ26AP > IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw== > cl...@cloud-emea.test.host[root@cloud-emea iso]# > > is the same key as I have in : > /var/cloudstack/management/.ssh/id_rsa.pub > > (For security reason I have replaced characters from ssh pub key) > > > I'm stuck here, I do not understand the logic. > > > Regards, > Cristian > > -Original Message----- > From: Pearl d'Silva > Sent: Friday, September 24, 2021 8:05 AM > To: users@cloudstack.apache.org > Subject: Re: Failed to authentication SSH user root on host - > repetitive error > > Hi Cristian, > > The exception you are seeing is most likely to happen if the > systemvm.iso hasn't been injected with the Management server's key. > One way to validate it, would be to go to your secondary store - I > presume you are working on a VMware setup - mount the > systemvm-4.15.2.iso and verify the authorized_keys data with what's > present on the MS at /var/cloudstack/management/.ssh/id_rsa.pub. > > Thanks, > Pearl > > > From: cristian.c@istream.today > Sent: Thursday, September 23, 2021 9:50 PM > To: users@cloudstack.apache.org > Subject: Failed to authentication SSH user root on host - repetitive > error > > Hello, > > > > > > I was not able to fix this error "Failed to authentication SSH" > and looks like a bug for me, I will explain here why. > > > > My setup before adding new ZONE: > > > > 1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This > is > an old setup, it was upgraded frequently, I think from 4.2 ) > 2. I add new VMware zone, but this one with Advanced Networking, I end > with this repetitive error and nothing else. > 3. I decide to add a new zone, the same as I have at (2) with Advanced > Networking, everything works perfect, no issue at systemvm deploy, > routers, instances. > 4. I delete the zone from point (2) > 5. I add again the same zone, end with the same error. > > > >I have tried multiple things, delete the template, add again, > inject, nothing works, I end with the same error, over and over, and I > do not see the why. > > > > > > 021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource] > (DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, > job-35231/job-35263, > cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to > Exception: > java.lang.Exception > > Message: Failed to authentication SSH user root on host 10.15.0.204 > > > > 2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb) > (logid:df4131be) Retrying after catching exception while trying to > secure agent for systemvm id=2025 > > com.cloud.utils.exception.CloudRuntimeException: Unable to > read/process > CSR: > Command failed due to Exception: java.lang.Exception > > Message: Failed to authentication SSH user root on host 10.15.0.204 > > > > > > Here is the full log : https://pastebin.com/raw/fFfmquVB > > > > > > Maybe someone have a hint for me . > > > > Regards, > > Cristian > > > > > > > > > >
Re: Failed to authentication SSH user root on host - repetitive error
Hi Cristian, Could you please check if the systemvm.iso on secondary storage has the public key ? # ls -l /var/cloudstack/mnt/32987294991135.77249d5c/systemvm/ total 92912 -rw-r--r--. 1 cloud cloud 95141888 Sep 15 15:44 systemvm-4.16.0.0-SNAPSHOT.iso secondary storage is mounted on mgmt server -Wei On Fri, 24 Sept 2021 at 09:51, wrote: > Hi Wei, > >I think here is the problem, when the key are injected, the SSVM is not > running yet, is starting few seconds later So, the answer is that I do > not have what to STOP, I can only delete the SSVM instance from VMware > side.. > > Thank you, > Cristian > > -Original Message- > From: Wei ZHOU > Sent: Friday, September 24, 2021 9:43 AM > To: users > Subject: Re: Failed to authentication SSH user root on host - repetitive > error > > Hi Cristian, > > When the new public key has been injected into systemvm.iso, please > stop/start or destroy SSVM. > > -Wei > > On Fri, 24 Sept 2021 at 08:16, wrote: > > > Hi Pearl, > > > > I already did a check via console into the secondary storage vm, > > I saw that there are not keys in authorized_keys. > > > > If I try to inject, I see this : /bin/bash > > /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh > > /var/cloudstack/management/.ssh/id_rsa.pub > > /var/cloudstack/management/.ssh/id_rsa > > /usr/share/cloudstack-common/vms/systemvm.iso > > mount: /dev/loop0 is write-protected, mounting read-only New public > > key is the same as the one in the systemvm.iso, not injecting it, not > > modifying systemvm.iso > > > > > > [root@cloud-emea systemvm]# cd /mnt/iso/ [root@cloud-emea iso]# ls > > agent.zip authorized_keys cloud-scripts.tgz [root@cloud-emea iso]# > > cat authorized_keys ssh-rsa > > > > B3NzaC1yc2EBIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZh > > cPueIl > > > > BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+xxx > > xx > > > > xx+Bd1Y2U1 > > xx+TxlYee > > > > +zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIM > > +tZ26AP > > IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw== > > cl...@cloud-emea.test.host[root@cloud-emea iso]# > > > > is the same key as I have in : > > /var/cloudstack/management/.ssh/id_rsa.pub > > > > (For security reason I have replaced characters from ssh pub key) > > > > > > I'm stuck here, I do not understand the logic. > > > > > > Regards, > > Cristian > > > > -Original Message- > > From: Pearl d'Silva > > Sent: Friday, September 24, 2021 8:05 AM > > To: users@cloudstack.apache.org > > Subject: Re: Failed to authentication SSH user root on host - > > repetitive error > > > > Hi Cristian, > > > > The exception you are seeing is most likely to happen if the > > systemvm.iso hasn't been injected with the Management server's key. > > One way to validate it, would be to go to your secondary store - I > > presume you are working on a VMware setup - mount the > > systemvm-4.15.2.iso and verify the authorized_keys data with what's > > present on the MS at /var/cloudstack/management/.ssh/id_rsa.pub. > > > > Thanks, > > Pearl > > > > > > From: cristian.c@istream.today > > Sent: Thursday, September 23, 2021 9:50 PM > > To: users@cloudstack.apache.org > > Subject: Failed to authentication SSH user root on host - repetitive > > error > > > > Hello, > > > > > > > > > > > > I was not able to fix this error "Failed to authentication SSH" > > and looks like a bug for me, I will explain here why. > > > > > > > > My setup before adding new ZONE: > > > > > > > > 1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This > > is > > an old setup, it was upgraded frequently, I think from 4.2 ) > > 2. I add new VMware zone, but this one with Advanced Networking, I > end > > with this repetitive error and nothing else. > > 3. I decide to add a new zone, the same as I have at (2) with > Advanced > > Networking, everything works perfect, no issue at systemvm deploy, > > routers, instances. > > 4. I delete the zone from point (2) > > 5. I add again the same zone, end with the same error. > > > >
RE: Failed to authentication SSH user root on host - repetitive error
Hi Pearl, I just did a SSVM redeploy in the same environment, but different zone and and I see the following differences : "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)" Working zone - https://pastebin.com/raw/88N6Fg6z "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB) datastore-6588","managed":"false","storagePort":"0","volumeSize":"(2.44 GB) 021-09-24 05:22:23,959 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-437:ctx-2f1519ba fra-de-001.test.host, job-25099/job-35371, cmd: StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the same as the cloned VM datastore UUID: fra-de-001 - 9d679070c7a6441592646bfbb82ef1b3 Not working zone - https://pastebin.com/raw/Um2uHgWM "isManaged":"false"}},"name":"ROOT-2056","size":"(0 bytes)"here is 0 bytes... datastore-6694","managed":"false","storagePort":"0","volumeSize":"(0 bytes) here is 0 bytes... 2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Mapped disk size is not the same as the cloned VM disk size: 0 - 262144( here I see this line, with disk size 0 ) 2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the same as the cloned VM datastore UUID: lnd-uk-002 - fb7b551a584e4b5192b8f371af56b303 I do not understand why I have here the size "0" Regards, Cristian -Original Message- From: Pearl d'Silva Sent: Friday, September 24, 2021 9:44 AM To: users@cloudstack.apache.org Subject: Re: Failed to authentication SSH user root on host - repetitive error By any chance, are there any old systemvm.isos in the secondary store, if yes, can you please delete them. Based on the information you've provided, it seems like, the key has been injected into systemvm.iso, but during boot up, a script (cloud-early-config) that sets up the VM before bootstrapping / patching isn't copying the auth key. Can you try the steps mentioned under the VMWare section of the doc: https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso and see if it helps. SystemVm.iso - Apache Cloudstack - Apache Software Foundation<https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.i so> Mechanism of propagating systemvm.iso. Systemvm.iso gets propagated differently for different hypervisors. It comes packaged as part of the Cloudstack rpm or is built on dev environments with the command "mvn clean install -P developer,systemvm" (More info on building cloudstack) .Once the iso is at the appropriate location depending on the hypervisor it is inserted as a cd drive and the ... cwiki.apache.org Thanks, Pearl From: cristian.c@istream.today Sent: Friday, September 24, 2021 11:45 AM To: users@cloudstack.apache.org Subject: RE: Failed to authentication SSH user root on host - repetitive error Hi Pearl, I already did a check via console into the secondary storage vm, I saw that there are not keys in authorized_keys. If I try to inject, I see this : /bin/bash /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh /var/cloudstack/management/.ssh/id_rsa.pub /var/cloudstack/management/.ssh/id_rsa /usr/share/cloudstack-common/vms/systemvm.iso mount: /dev/loop0 is write-protected, mounting read-only New public key is the same as the one in the systemvm.iso, not injecting it, not modifying systemvm.iso [root@cloud-emea systemvm]# cd /mnt/iso/ [root@cloud-emea iso]# ls agent.zip authorized_keys cloud-scripts.tgz [root@cloud-emea iso]# cat authorized_keys ssh-rsa B3NzaC1yc2EBIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZhcPueIl BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+x xx+Bd1Y2U1Tx xx+lYee +zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIMtZ +26AP IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw== cl...@cloud-emea.test.host[root@cloud-emea iso]# is the same key as I have in : /var/cloudstack/management/.ssh/id_rsa.pub (For security reason I have replaced characters from ssh pub key) I'm stuck here, I do not understand the logic. Regards, Cristian -Original Message- From: Pearl d'Silva Sent: Friday, September 24, 2021 8:05 AM To: users@cloudstack.apache.org Subject: Re: Failed to authentication SSH user root on host - repetitive error Hi Cristian
Re: Failed to authentication SSH user root on host - repetitive error
Hi Cristian, After going through your earlier mails again, and correct me if I am wrong, but I noticed in one of your mails you mentioned that the env that's causing issues was upgraded from a lower version, right? So, ideally when doing upgrades, prior to initiating the upgrade we first take a backup of the DB, register the template (new systemVM template) following this https://docs.cloudstack.apache.org/en/latest/upgrading/upgrade/upgrade-4.15.html#update-system-vm-templates and then upgrade the environment. However, you mentioned that you used the cloud-install-sys-tmplt script to seed and register the template. I am not familiar if using that script during upgrades works (it may, but I haven't ever used that approach). If it is possible, it would be advisable to rollback your env to the earlier version (disable your zone, destroy the systemVMs before that) and then follow the upgrade steps as mentioned in the documentation: https://docs.cloudstack.apache.org/en/latest/upgrading/index.html Thanks, Pearl From: cristian.c@istream.today Sent: Friday, September 24, 2021 3:08 PM To: users@cloudstack.apache.org Subject: RE: Failed to authentication SSH user root on host - repetitive error Hi Pearl, I just did a SSVM redeploy in the same environment, but different zone and and I see the following differences : "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)" Working zone - https://pastebin.com/raw/88N6Fg6z "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB) datastore-6588","managed":"false","storagePort":"0","volumeSize":"(2.44 GB) 021-09-24 05:22:23,959 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-437:ctx-2f1519ba fra-de-001.test.host, job-25099/job-35371, cmd: StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the same as the cloned VM datastore UUID: fra-de-001 - 9d679070c7a6441592646bfbb82ef1b3 Not working zone - https://pastebin.com/raw/Um2uHgWM "isManaged":"false"}},"name":"ROOT-2056","size":"(0 bytes)"here is 0 bytes... datastore-6694","managed":"false","storagePort":"0","volumeSize":"(0 bytes) here is 0 bytes... 2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Mapped disk size is not the same as the cloned VM disk size: 0 - 262144( here I see this line, with disk size 0 ) 2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd: StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the same as the cloned VM datastore UUID: lnd-uk-002 - fb7b551a584e4b5192b8f371af56b303 I do not understand why I have here the size "0" Regards, Cristian -Original Message- From: Pearl d'Silva Sent: Friday, September 24, 2021 9:44 AM To: users@cloudstack.apache.org Subject: Re: Failed to authentication SSH user root on host - repetitive error By any chance, are there any old systemvm.isos in the secondary store, if yes, can you please delete them. Based on the information you've provided, it seems like, the key has been injected into systemvm.iso, but during boot up, a script (cloud-early-config) that sets up the VM before bootstrapping / patching isn't copying the auth key. Can you try the steps mentioned under the VMWare section of the doc: https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso and see if it helps. SystemVm.iso - Apache Cloudstack - Apache Software Foundation<https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.i so> Mechanism of propagating systemvm.iso. Systemvm.iso gets propagated differently for different hypervisors. It comes packaged as part of the Cloudstack rpm or is built on dev environments with the command "mvn clean install -P developer,systemvm" (More info on building cloudstack) .Once the iso is at the appropriate location depending on the hypervisor it is inserted as a cd drive and the ... cwiki.apache.org Thanks, Pearl From: cristian.c@istream.today Sent: Friday, September 24, 2021 11:45 AM To: users@cloudstack.apache.org Subject: RE: Failed to authentication SSH user root on host - repetitive error Hi Pearl, I already did a check via console into the secondary storage vm, I saw that there are not keys in authorized_keys. If I try to inject, I see this : /bin/bash /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh /var/cloudstack/management/.ssh/id_rsa.pub /var/cloudstack/management/.ssh/id_rsa /usr/share/cloudstack-com
Re: Failed to authentication SSH user root on host - repetitive error
Hi Pearl, I did upgrades multiple times, I always add the template before upgrade. If was something related to this, i was expecting not to work for all zones, but right now is working for 2 zones from the same setup, environment. Regards, Cristian On Fri, Sep 24, 2021, 14:40 Pearl d'Silva wrote: > Hi Cristian, > > After going through your earlier mails again, and correct me if I am > wrong, but I noticed in one of your mails you mentioned that the env that's > causing issues was upgraded from a lower version, right? So, ideally when > doing upgrades, prior to initiating the upgrade we first take a backup of > the DB, register the template (new systemVM template) following this > https://docs.cloudstack.apache.org/en/latest/upgrading/upgrade/upgrade-4.15.html#update-system-vm-templates > and then upgrade the environment. However, you mentioned that you used the > cloud-install-sys-tmplt script to seed and register the template. I am not > familiar if using that script during upgrades works (it may, but I haven't > ever used that approach). > If it is possible, it would be advisable to rollback your env to the > earlier version (disable your zone, destroy the systemVMs before that) and > then follow the upgrade steps as mentioned in the documentation: > https://docs.cloudstack.apache.org/en/latest/upgrading/index.html > > Thanks, > Pearl > > > > From: cristian.c@istream.today > Sent: Friday, September 24, 2021 3:08 PM > To: users@cloudstack.apache.org > Subject: RE: Failed to authentication SSH user root on host - repetitive > error > > Hi Pearl, > > I just did a SSVM redeploy in the same environment, but different zone and > and I see the following differences : > > "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)" > > > Working zone - https://pastebin.com/raw/88N6Fg6z > > "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB) > > datastore-6588","managed":"false","storagePort":"0","volumeSize":"(2.44 > GB) > > 021-09-24 05:22:23,959 INFO [c.c.h.v.r.VmwareResource] > (DirectAgent-437:ctx-2f1519ba fra-de-001.test.host, job-25099/job-35371, > cmd: StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the > same as the cloned VM datastore UUID: fra-de-001 - > 9d679070c7a6441592646bfbb82ef1b3 > > > > Not working zone - https://pastebin.com/raw/Um2uHgWM > > "isManaged":"false"}},"name":"ROOT-2056","size":"(0 bytes)"here is 0 > bytes... > > datastore-6694","managed":"false","storagePort":"0","volumeSize":"(0 > bytes) > here is 0 bytes... > > 2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource] > (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, > cmd: > StartCommand) (logid:d442432d) Mapped disk size is not the same as the > cloned VM disk size: 0 - 262144( here I see this line, with disk > size 0 ) > > 2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource] > (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, > cmd: > StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the same > as > the cloned VM datastore UUID: lnd-uk-002 - fb7b551a584e4b5192b8f371af56b303 > > > > I do not understand why I have here the size "0" > > > Regards, > Cristian > > > > > -Original Message- > From: Pearl d'Silva > Sent: Friday, September 24, 2021 9:44 AM > To: users@cloudstack.apache.org > Subject: Re: Failed to authentication SSH user root on host - repetitive > error > > By any chance, are there any old systemvm.isos in the secondary store, if > yes, can you please delete them. Based on the information you've provided, > it seems like, the key has been injected into systemvm.iso, but during boot > up, a script (cloud-early-config) that sets up the VM before bootstrapping > / > patching isn't copying the auth key. > Can you try the steps mentioned under the VMWare section of the doc: > https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso > and see if it helps. > SystemVm.iso - Apache Cloudstack - Apache Software > Foundation< > https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.i > so> > Mechanism of propagating systemvm.iso. Systemvm.iso gets propagated > differently for different hypervisors. It comes packaged as part of the > Cloudstack rpm or is built on dev environments with the command &qu
Re: Failed to authentication SSH user root on host - repetitive error
Hi Crisitian, To my understanding, you are trying to add a new zone and getting issues with the system VM having some SSH failures. May I ask you to start everything fresh? I mean to prepare new secondary storage, seed the system VM template into new secondary storage and then deploy the zone from the beginning of the zone creation wizard. Make sure your management server has the latest packages and systemvm.iso is also as expected to the CloudStack version you are trying. Thanks, Harikrishna From: Cristian Ciobanu Sent: Friday, September 24, 2021 5:19 PM To: users@cloudstack.apache.org Subject: Re: Failed to authentication SSH user root on host - repetitive error Hi Pearl, I did upgrades multiple times, I always add the template before upgrade. If was something related to this, i was expecting not to work for all zones, but right now is working for 2 zones from the same setup, environment. Regards, Cristian On Fri, Sep 24, 2021, 14:40 Pearl d'Silva wrote: > Hi Cristian, > > After going through your earlier mails again, and correct me if I am > wrong, but I noticed in one of your mails you mentioned that the env that's > causing issues was upgraded from a lower version, right? So, ideally when > doing upgrades, prior to initiating the upgrade we first take a backup of > the DB, register the template (new systemVM template) following this > https://docs.cloudstack.apache.org/en/latest/upgrading/upgrade/upgrade-4.15.html#update-system-vm-templates > and then upgrade the environment. However, you mentioned that you used the > cloud-install-sys-tmplt script to seed and register the template. I am not > familiar if using that script during upgrades works (it may, but I haven't > ever used that approach). > If it is possible, it would be advisable to rollback your env to the > earlier version (disable your zone, destroy the systemVMs before that) and > then follow the upgrade steps as mentioned in the documentation: > https://docs.cloudstack.apache.org/en/latest/upgrading/index.html > > Thanks, > Pearl > > > > From: cristian.c@istream.today > Sent: Friday, September 24, 2021 3:08 PM > To: users@cloudstack.apache.org > Subject: RE: Failed to authentication SSH user root on host - repetitive > error > > Hi Pearl, > > I just did a SSVM redeploy in the same environment, but different zone and > and I see the following differences : > > "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)" > > > Working zone - https://pastebin.com/raw/88N6Fg6z > > "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB) > > datastore-6588","managed":"false","storagePort":"0","volumeSize":"(2.44 > GB) > > 021-09-24 05:22:23,959 INFO [c.c.h.v.r.VmwareResource] > (DirectAgent-437:ctx-2f1519ba fra-de-001.test.host, job-25099/job-35371, > cmd: StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the > same as the cloned VM datastore UUID: fra-de-001 - > 9d679070c7a6441592646bfbb82ef1b3 > > > > Not working zone - https://pastebin.com/raw/Um2uHgWM > > "isManaged":"false"}},"name":"ROOT-2056","size":"(0 bytes)"here is 0 > bytes... > > datastore-6694","managed":"false","storagePort":"0","volumeSize":"(0 > bytes) > here is 0 bytes... > > 2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource] > (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, > cmd: > StartCommand) (logid:d442432d) Mapped disk size is not the same as the > cloned VM disk size: 0 - 262144( here I see this line, with disk > size 0 ) > > 2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource] > (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, > cmd: > StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the same > as > the cloned VM datastore UUID: lnd-uk-002 - fb7b551a584e4b5192b8f371af56b303 > > > > I do not understand why I have here the size "0" > > > Regards, > Cristian > > > > > -Original Message- > From: Pearl d'Silva > Sent: Friday, September 24, 2021 9:44 AM > To: users@cloudstack.apache.org > Subject: Re: Failed to authentication SSH user root on host - repetitive > error > > By any chance, are there any old systemvm.isos in the secondary store, if > yes, can you please delete them. Based on the information you've provided, > it seems like, the key has been injected into
RE: Failed to authentication SSH user root on host - repetitive error
Hello, I did this in 3 different new Setup (new Management install, NFS, etc), no matter if I add as first zone or second zone. " I have tested this with multiple Cloudstack versions, 4.15.0, 4.15.1, 4.15.2 and VMware, 6.5 and with 6.7 different patches and with the last patches. I'm 100% that is related to these 2 servers, I do not understand what is wrong with these servers, this is the problem. We have 2 identical servers' hardware/ESXI, the only difference is the location, the not working, are in UK, and any other location we have, DE, NYC, FR, CA, etc. works fine. But we tested with the exact same servers which are from DE(Germany) works perfect, with any ACS or VMware version (also same CIDR range/size for public and private), when we test with UK, we get same error no matter what." I want to mention that we have these servers (UK, DE, FR, CA ) from OVH in vRack and the UK, are not working. ( I manage these server for more than 5 years ) Regards, Cristian -Original Message- From: Harikrishna Patnala Sent: Monday, September 27, 2021 1:29 PM To: users@cloudstack.apache.org Subject: Re: Failed to authentication SSH user root on host - repetitive error Hi Crisitian, To my understanding, you are trying to add a new zone and getting issues with the system VM having some SSH failures. May I ask you to start everything fresh? I mean to prepare new secondary storage, seed the system VM template into new secondary storage and then deploy the zone from the beginning of the zone creation wizard. Make sure your management server has the latest packages and systemvm.iso is also as expected to the CloudStack version you are trying. Thanks, Harikrishna From: Cristian Ciobanu Sent: Friday, September 24, 2021 5:19 PM To: users@cloudstack.apache.org Subject: Re: Failed to authentication SSH user root on host - repetitive error Hi Pearl, I did upgrades multiple times, I always add the template before upgrade. If was something related to this, i was expecting not to work for all zones, but right now is working for 2 zones from the same setup, environment. Regards, Cristian On Fri, Sep 24, 2021, 14:40 Pearl d'Silva wrote: > Hi Cristian, > > After going through your earlier mails again, and correct me if I am > wrong, but I noticed in one of your mails you mentioned that the env > that's causing issues was upgraded from a lower version, right? So, > ideally when doing upgrades, prior to initiating the upgrade we first > take a backup of the DB, register the template (new systemVM > template) following this > https://docs.cloudstack.apache.org/en/latest/upgrading/upgrade/upgrade > -4.15.html#update-system-vm-templates > and then upgrade the environment. However, you mentioned that you used > the cloud-install-sys-tmplt script to seed and register the template. > I am not familiar if using that script during upgrades works (it may, > but I haven't ever used that approach). > If it is possible, it would be advisable to rollback your env to the > earlier version (disable your zone, destroy the systemVMs before that) > and then follow the upgrade steps as mentioned in the documentation: > https://docs.cloudstack.apache.org/en/latest/upgrading/index.html > > Thanks, > Pearl > > > > From: cristian.c@istream.today > Sent: Friday, September 24, 2021 3:08 PM > To: users@cloudstack.apache.org > Subject: RE: Failed to authentication SSH user root on host - > repetitive error > > Hi Pearl, > > I just did a SSVM redeploy in the same environment, but different zone > and and I see the following differences : > > "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)" > > > Working zone - https://pastebin.com/raw/88N6Fg6z > > "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB) > > > datastore-6588","managed":"false","storagePort":"0","volumeSize":"(2.4 > 4 > GB) > > 021-09-24 05:22:23,959 INFO [c.c.h.v.r.VmwareResource] > (DirectAgent-437:ctx-2f1519ba fra-de-001.test.host, > job-25099/job-35371, > cmd: StartCommand) (logid:d442432d) Mapped disk datastore UUID is not > the same as the cloned VM datastore UUID: fra-de-001 - > 9d679070c7a6441592646bfbb82ef1b3 > > > > Not working zone - https://pastebin.com/raw/Um2uHgWM > > "isManaged":"false"}},"name":"ROOT-2056","size":"(0 bytes)"here is 0 > bytes... > > datastore-6694","managed":"false","storagePort":"0","volumeSize":"(0 &g