You guys can check job-1248274 logs or SSVM s-25457-VM that has successfully connected agent state.
Ammad Ali On Mon, Sep 14, 2020 at 2:33 PM Ammad Syed <syedamma...@gmail.com> wrote: > Here are the upgrade management server logs. > > > https://drive.google.com/file/d/17cxh7f-24ibnXCKvUUPqt3p1YTzlMQN8/view?usp=sharing > > Ammad Ali > > On Mon, Sep 14, 2020 at 2:17 PM Ammad Syed <syedamma...@gmail.com> wrote: > >> In addition to previous email there is only one host in one zone where >> systemVM agent goes up and on all other hosts on that zone agent failed. >> >> If you guys need, I can provide management server logs as well. >> >> Also is there a way to enable debugging in ACS logs to specifically find >> out where the problem is? >> >> Ammad Ali >> >> On Mon, Sep 14, 2020 at 10:39 AM Ammad Syed <syedamma...@gmail.com> >> wrote: >> >>> Hi Perl, >>> >>> I have taken those steps and verified that systemvm.iso is copied to all >>> hosts in all zones. >>> >>> I have recreated the systemvm and ssh to that host and checked the >>> md5sum of iso there and on acs. Both were same. >>> >>> However the md5sum on which systemvm was working has also same md5sum of >>> systemvm iso. The iso is getting copied successfully. The problem looks >>> somewhere else. >>> >>> I have checked in xenserver logs as well but didn’t find any logs that >>> something has failed. >>> >>> Ammad >>> Sent from my iPhone >>> >>> > On 14-Sep-2020, at 9:15 AM, Pearl d'Silva <pearl.dsi...@shapeblue.com> >>> wrote: >>> > >>> > Hi Ammad, >>> > >>> > Is the understanding right that the steps as mentioned by you in the >>> previous mail has in-fact worked on one zone? If that's the case, could you >>> please ensure that all the hosts in all the other zones have the new >>> systemVM iso copied into them by checking the timestamps as well and >>> comparing the checksums against the iso on the management server, so that >>> when the system VM's are recreated, they pick up the new iso. >>> > >>> > Thanks, >>> > Pearl >>> > >>> > ________________________________ >>> > From: Ammad Syed <syedamma...@gmail.com> >>> > Sent: Sunday, September 13, 2020 12:49 PM >>> > To: users@cloudstack.apache.org <users@cloudstack.apache.org> >>> > Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error >>> > >>> > Hi Andrija, >>> > >>> > Here is the permission of mount point in SSVM. >>> > >>> > root@s-25437-VM:~# df -h >>> > Filesystem Size Used Avail Use% Mounted on >>> > udev 233M 0 233M 0% /dev >>> > tmpfs 98M 19M 80M 19% /run >>> > /dev/xvda5 1.1G 773M 282M 74% / >>> > tmpfs 244M 0 244M 0% /dev/shm >>> > tmpfs 5.0M 0 5.0M 0% /run/lock >>> > tmpfs 244M 0 244M 0% /sys/fs/cgroup >>> > /dev/xvda1 92M 35M 57M 39% /boot >>> > /dev/xvda6 435M 21M 410M 5% /var >>> > /dev/xvda7 75M 1.6M 72M 3% /tmp >>> > 172.16.10.35:/nfs/KHI02 12T 7.0T 5.1T 59% >>> > /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092 >>> > tmpfs 49M 0 49M 0% /run/user/0 >>> > root@s-25437-VM:~# >>> > root@s-25437-VM:~# >>> > root@s-25437-VM:~# cd >>> /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092 >>> > root@s-25437-VM:/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092# >>> > root@s-25437-VM:/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092# >>> ls >>> > -lah >>> > total 12K >>> > drwxrwxrwx 5 root root 70 Dec 21 2018 . >>> > drwxrwxrwx 3 root root 4.0K Sep 10 23:12 .. >>> > drwxrwxrwx 52 root root 4.0K Aug 13 19:37 snapshots >>> > drwxrwxrwx 3 root root 26 Jun 7 2013 template >>> > drwxrwxrwx 98 root root 4.0K Sep 1 12:52 volumes >>> > >>> > -Ammad Ali >>> > >>> >> On Fri, Sep 11, 2020 at 6:09 PM Andrija Panic < >>> andrija.pa...@gmail.com> >>> >> wrote: >>> >> >>> >> Can you share permissions of your secondary storage (mount it then ls >>> -lah >>> >> the mount point) >>> >> >>> > >>> > pearl.dsi...@shapeblue.com >>> > www.shapeblue.com >>> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK >>> > @shapeblue >>> > >>> > >>> > >>> >>> On Fri, 11 Sep 2020 at 01:39, Ammad Syed <syedamma...@gmail.com> >>> wrote: >>> >>> >>> >>> Hi Andrija, >>> >>> >>> >>> I have performed an upgrade on my production system from 4.11.3 to >>> >> 4.13.1. >>> >>> Even I cleared the tags but the issue is still there. >>> >>> >>> >>> I have four zones and only in one zone and on specific host, the >>> >>> systemVM's agent goes up but on all other zones, key injection to >>> >> systemVM >>> >>> is still failing on all zones and PODs. I have checked, the updated >>> ISO >>> >> is >>> >>> there on all hosts. The md5sum of systemvm.iso is same on xen hosts >>> and >>> >> acs >>> >>> host. >>> >>> >>> >>> Look like a weird problem. How can I troubleshoot this further ? Any >>> >> advise >>> >>> would be appreciated. >>> >>> >>> >>> -Ammad >>> >>> >>> >> >>> >> >>> >> -- >>> >> >>> >> Andrija Panić >>> >> >>> > >>> > >>> > -- >>> > Regards, >>> > >>> > >>> > Syed Ammad Ali >>> >> >> >> -- >> Regards, >> >> >> Syed Ammad Ali >> > > > -- > Regards, > > > Syed Ammad Ali > -- Regards, Syed Ammad Ali
