Ammad, what's you current status with your issue?
In logs I can see that there is some issue with SR: 2020-09-10 22:58:48,374 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper] (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) 1. The VM s-25505-VM is in Starting state. 2020-09-10 22:58:48,375 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) no guest OS type, start it as HVM guest 2020-09-10 22:58:48,390 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) Created VM 3fa168bd-9e29-03df-52bb-a0fe2a53d390 for s-25505-VM 2020-09-10 22:58:48,394 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) PV args are %template=domP%type=secstorage%host=172.16.2.42%port=8250%name=s-25505-VM%zone=10%pod=10%guid=s-25505-VM%workers=5%resource=com.cloud.storage.resource.PremiumSecondaryStorageResource%instance=SecStorage%sslcopy=false%role=templateProcessor%mtu=1500%eth2ip=175.107.206.196%eth2mask=255.255.255.0%gateway=175.107.206.1%public.network.device=eth2%eth0ip=169.254.1.184%eth0mask=255.255.0.0%eth1ip=172.16.2.56%eth1mask=255.255.255.192%mgmtcidr= 172.16.0.0/26%localgw=172.16.2.62%private.network.device=eth1%internaldns1=202.163.96.3%internaldns2=202.163.96.4%dns1=202.163.96.3%dns2=202.163.96.4%nfsVersion=null 2020-09-10 22:58:48,396 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) HVM args are template=domP type=secstorage host=172.16.2.42 port=8250 name=s-25505-VM zone=10 pod=10 guid=s-25505-VM workers=5 resource=com.cloud.storage.resource.PremiumSecondaryStorageResource instance=SecStorage sslcopy=false role=templateProcessor mtu=1500 eth2ip=175.107.206.196 eth2mask=255.255.255.0 gateway=175.107.206.1 public.network.device=eth2 eth0ip=169.254.1.184 eth0mask=255.255.0.0 eth1ip=172.16.2.56 eth1mask=255.255.255.192 mgmtcidr=172.16.0.0/26 localgw=172.16.2.62 private.network.device=eth1 internaldns1=202.163.96.3 internaldns2=202.163.96.4 dns1=202.163.96.3 dns2=202.163.96.4 nfsVersion=null 2020-09-10 22:58:48,399 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) Failed to find SR by name 'XenServer Tools', will try to find 'XCP-ng Tools' SR 2020-09-10 22:58:48,400 WARN [c.c.h.x.r.w.x.CitrixStartCommandWrapper] (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) Catch Exception: class com.cloud.utils.exception.CloudRuntimeException due to com.cloud.utils.exception.CloudRuntimeException: There are 0 SRs with name XenServer Tools com.cloud.utils.exception.CloudRuntimeException: There are 0 SRs with name XenServer Tools at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.createPatchVbd(CitrixResourceBase.java:1061) Your SSVM can not even start, leave alone trying to access it etc. Also, after the upgrade, I saw that the public SSH key was successfully injected into the systemvm.iso, and that ISO is copied to all hosts, as well as the current id_rsa private key - so assuming your SSVM starts sucessfully, you should be able to access it, as a) you have the id_rsa on each XS host, and b) systemvm.iso does contain a good public_key(inside the authorized_keys file) - so assuming netwotk is UP (169.254...), you should be able to access it via ssh on port 3922. Best, On Mon, 14 Sep 2020 at 11:39, Ammad Syed <syedamma...@gmail.com> wrote: > You guys can check job-1248274 logs or SSVM s-25457-VM that has > successfully connected agent state. > > Ammad Ali > > On Mon, Sep 14, 2020 at 2:33 PM Ammad Syed <syedamma...@gmail.com> wrote: > > > Here are the upgrade management server logs. > > > > > > > https://drive.google.com/file/d/17cxh7f-24ibnXCKvUUPqt3p1YTzlMQN8/view?usp=sharing > > > > Ammad Ali > > > > On Mon, Sep 14, 2020 at 2:17 PM Ammad Syed <syedamma...@gmail.com> > wrote: > > > >> In addition to previous email there is only one host in one zone where > >> systemVM agent goes up and on all other hosts on that zone agent failed. > >> > >> If you guys need, I can provide management server logs as well. > >> > >> Also is there a way to enable debugging in ACS logs to specifically find > >> out where the problem is? > >> > >> Ammad Ali > >> > >> On Mon, Sep 14, 2020 at 10:39 AM Ammad Syed <syedamma...@gmail.com> > >> wrote: > >> > >>> Hi Perl, > >>> > >>> I have taken those steps and verified that systemvm.iso is copied to > all > >>> hosts in all zones. > >>> > >>> I have recreated the systemvm and ssh to that host and checked the > >>> md5sum of iso there and on acs. Both were same. > >>> > >>> However the md5sum on which systemvm was working has also same md5sum > of > >>> systemvm iso. The iso is getting copied successfully. The problem looks > >>> somewhere else. > >>> > >>> I have checked in xenserver logs as well but didn’t find any logs that > >>> something has failed. > >>> > >>> Ammad > >>> Sent from my iPhone > >>> > >>> > On 14-Sep-2020, at 9:15 AM, Pearl d'Silva < > pearl.dsi...@shapeblue.com> > >>> wrote: > >>> > > >>> > Hi Ammad, > >>> > > >>> > Is the understanding right that the steps as mentioned by you in the > >>> previous mail has in-fact worked on one zone? If that's the case, > could you > >>> please ensure that all the hosts in all the other zones have the new > >>> systemVM iso copied into them by checking the timestamps as well and > >>> comparing the checksums against the iso on the management server, so > that > >>> when the system VM's are recreated, they pick up the new iso. > >>> > > >>> > Thanks, > >>> > Pearl > >>> > > >>> > ________________________________ > >>> > From: Ammad Syed <syedamma...@gmail.com> > >>> > Sent: Sunday, September 13, 2020 12:49 PM > >>> > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > >>> > Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error > >>> > > >>> > Hi Andrija, > >>> > > >>> > Here is the permission of mount point in SSVM. > >>> > > >>> > root@s-25437-VM:~# df -h > >>> > Filesystem Size Used Avail Use% Mounted on > >>> > udev 233M 0 233M 0% /dev > >>> > tmpfs 98M 19M 80M 19% /run > >>> > /dev/xvda5 1.1G 773M 282M 74% / > >>> > tmpfs 244M 0 244M 0% /dev/shm > >>> > tmpfs 5.0M 0 5.0M 0% /run/lock > >>> > tmpfs 244M 0 244M 0% /sys/fs/cgroup > >>> > /dev/xvda1 92M 35M 57M 39% /boot > >>> > /dev/xvda6 435M 21M 410M 5% /var > >>> > /dev/xvda7 75M 1.6M 72M 3% /tmp > >>> > 172.16.10.35:/nfs/KHI02 12T 7.0T 5.1T 59% > >>> > /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092 > >>> > tmpfs 49M 0 49M 0% /run/user/0 > >>> > root@s-25437-VM:~# > >>> > root@s-25437-VM:~# > >>> > root@s-25437-VM:~# cd > >>> /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092 > >>> > root@s-25437-VM > :/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092# > >>> > root@s-25437-VM > :/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092# > >>> ls > >>> > -lah > >>> > total 12K > >>> > drwxrwxrwx 5 root root 70 Dec 21 2018 . > >>> > drwxrwxrwx 3 root root 4.0K Sep 10 23:12 .. > >>> > drwxrwxrwx 52 root root 4.0K Aug 13 19:37 snapshots > >>> > drwxrwxrwx 3 root root 26 Jun 7 2013 template > >>> > drwxrwxrwx 98 root root 4.0K Sep 1 12:52 volumes > >>> > > >>> > -Ammad Ali > >>> > > >>> >> On Fri, Sep 11, 2020 at 6:09 PM Andrija Panic < > >>> andrija.pa...@gmail.com> > >>> >> wrote: > >>> >> > >>> >> Can you share permissions of your secondary storage (mount it then > ls > >>> -lah > >>> >> the mount point) > >>> >> > >>> > > >>> > pearl.dsi...@shapeblue.com > >>> > www.shapeblue.com > >>> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > >>> > @shapeblue > >>> > > >>> > > >>> > > >>> >>> On Fri, 11 Sep 2020 at 01:39, Ammad Syed <syedamma...@gmail.com> > >>> wrote: > >>> >>> > >>> >>> Hi Andrija, > >>> >>> > >>> >>> I have performed an upgrade on my production system from 4.11.3 to > >>> >> 4.13.1. > >>> >>> Even I cleared the tags but the issue is still there. > >>> >>> > >>> >>> I have four zones and only in one zone and on specific host, the > >>> >>> systemVM's agent goes up but on all other zones, key injection to > >>> >> systemVM > >>> >>> is still failing on all zones and PODs. I have checked, the updated > >>> ISO > >>> >> is > >>> >>> there on all hosts. The md5sum of systemvm.iso is same on xen hosts > >>> and > >>> >> acs > >>> >>> host. > >>> >>> > >>> >>> Look like a weird problem. How can I troubleshoot this further ? > Any > >>> >> advise > >>> >>> would be appreciated. > >>> >>> > >>> >>> -Ammad > >>> >>> > >>> >> > >>> >> > >>> >> -- > >>> >> > >>> >> Andrija Panić > >>> >> > >>> > > >>> > > >>> > -- > >>> > Regards, > >>> > > >>> > > >>> > Syed Ammad Ali > >>> > >> > >> > >> -- > >> Regards, > >> > >> > >> Syed Ammad Ali > >> > > > > > > -- > > Regards, > > > > > > Syed Ammad Ali > > > > > -- > Regards, > > > Syed Ammad Ali > -- Andrija Panić
