Here are the upgrade management server logs. https://drive.google.com/file/d/17cxh7f-24ibnXCKvUUPqt3p1YTzlMQN8/view?usp=sharing
Ammad Ali On Mon, Sep 14, 2020 at 2:17 PM Ammad Syed <syedamma...@gmail.com> wrote: > In addition to previous email there is only one host in one zone where > systemVM agent goes up and on all other hosts on that zone agent failed. > > If you guys need, I can provide management server logs as well. > > Also is there a way to enable debugging in ACS logs to specifically find > out where the problem is? > > Ammad Ali > > On Mon, Sep 14, 2020 at 10:39 AM Ammad Syed <syedamma...@gmail.com> wrote: > >> Hi Perl, >> >> I have taken those steps and verified that systemvm.iso is copied to all >> hosts in all zones. >> >> I have recreated the systemvm and ssh to that host and checked the md5sum >> of iso there and on acs. Both were same. >> >> However the md5sum on which systemvm was working has also same md5sum of >> systemvm iso. The iso is getting copied successfully. The problem looks >> somewhere else. >> >> I have checked in xenserver logs as well but didn’t find any logs that >> something has failed. >> >> Ammad >> Sent from my iPhone >> >> > On 14-Sep-2020, at 9:15 AM, Pearl d'Silva <pearl.dsi...@shapeblue.com> >> wrote: >> > >> > Hi Ammad, >> > >> > Is the understanding right that the steps as mentioned by you in the >> previous mail has in-fact worked on one zone? If that's the case, could you >> please ensure that all the hosts in all the other zones have the new >> systemVM iso copied into them by checking the timestamps as well and >> comparing the checksums against the iso on the management server, so that >> when the system VM's are recreated, they pick up the new iso. >> > >> > Thanks, >> > Pearl >> > >> > ________________________________ >> > From: Ammad Syed <syedamma...@gmail.com> >> > Sent: Sunday, September 13, 2020 12:49 PM >> > To: users@cloudstack.apache.org <users@cloudstack.apache.org> >> > Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error >> > >> > Hi Andrija, >> > >> > Here is the permission of mount point in SSVM. >> > >> > root@s-25437-VM:~# df -h >> > Filesystem Size Used Avail Use% Mounted on >> > udev 233M 0 233M 0% /dev >> > tmpfs 98M 19M 80M 19% /run >> > /dev/xvda5 1.1G 773M 282M 74% / >> > tmpfs 244M 0 244M 0% /dev/shm >> > tmpfs 5.0M 0 5.0M 0% /run/lock >> > tmpfs 244M 0 244M 0% /sys/fs/cgroup >> > /dev/xvda1 92M 35M 57M 39% /boot >> > /dev/xvda6 435M 21M 410M 5% /var >> > /dev/xvda7 75M 1.6M 72M 3% /tmp >> > 172.16.10.35:/nfs/KHI02 12T 7.0T 5.1T 59% >> > /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092 >> > tmpfs 49M 0 49M 0% /run/user/0 >> > root@s-25437-VM:~# >> > root@s-25437-VM:~# >> > root@s-25437-VM:~# cd >> /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092 >> > root@s-25437-VM:/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092# >> > root@s-25437-VM:/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092# >> ls >> > -lah >> > total 12K >> > drwxrwxrwx 5 root root 70 Dec 21 2018 . >> > drwxrwxrwx 3 root root 4.0K Sep 10 23:12 .. >> > drwxrwxrwx 52 root root 4.0K Aug 13 19:37 snapshots >> > drwxrwxrwx 3 root root 26 Jun 7 2013 template >> > drwxrwxrwx 98 root root 4.0K Sep 1 12:52 volumes >> > >> > -Ammad Ali >> > >> >> On Fri, Sep 11, 2020 at 6:09 PM Andrija Panic <andrija.pa...@gmail.com >> > >> >> wrote: >> >> >> >> Can you share permissions of your secondary storage (mount it then ls >> -lah >> >> the mount point) >> >> >> > >> > pearl.dsi...@shapeblue.com >> > www.shapeblue.com >> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK >> > @shapeblue >> > >> > >> > >> >>> On Fri, 11 Sep 2020 at 01:39, Ammad Syed <syedamma...@gmail.com> >> wrote: >> >>> >> >>> Hi Andrija, >> >>> >> >>> I have performed an upgrade on my production system from 4.11.3 to >> >> 4.13.1. >> >>> Even I cleared the tags but the issue is still there. >> >>> >> >>> I have four zones and only in one zone and on specific host, the >> >>> systemVM's agent goes up but on all other zones, key injection to >> >> systemVM >> >>> is still failing on all zones and PODs. I have checked, the updated >> ISO >> >> is >> >>> there on all hosts. The md5sum of systemvm.iso is same on xen hosts >> and >> >> acs >> >>> host. >> >>> >> >>> Look like a weird problem. How can I troubleshoot this further ? Any >> >> advise >> >>> would be appreciated. >> >>> >> >>> -Ammad >> >>> >> >> >> >> >> >> -- >> >> >> >> Andrija Panić >> >> >> > >> > >> > -- >> > Regards, >> > >> > >> > Syed Ammad Ali >> > > > -- > Regards, > > > Syed Ammad Ali > -- Regards, Syed Ammad Ali
