Glad to see you solved the issue! On Mon, 7 Sep 2020 at 10:42, Ammad Syed <syedamma...@gmail.com> wrote:
> Hi, > > Thank you guys, this has fixed my problem. Here are the steps that I have > performed. > > - Did a fresh installation of 4.13.1 and copied its systemvm.iso on my ACS > in /usr/share/cloudstack-common/vms/. > - Shutdown management server. > - Deleted systemVMs from xencenter. > - Executed key injection script > "/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh > /root/.ssh/id_rsa.pub /root/.ssh/id_rsa > /usr/share/cloudstack-common/vms/systemvm.iso" > - Cleared the tags xe host-param-clear param-name=tags uuid=<uuid of the XS > host> > - Started management server. > - Recreated systemVMs. > > Thank you guys for your support. > > - Ammad > > On Thu, Sep 3, 2020 at 11:40 AM Andrija Panic <andrija.pa...@gmail.com> > wrote: > > > Ammad, > > > > please check what Pearl has suggested, but I think the correct syntax > > (different id_rsa,pub, specifically the one used by ACS) should be as > > following (you can always grep your older logs for "injectkey.sh" : > > > > /bin/bash /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh > > /var/cloudstack/management/.ssh/id_rsa.pub > > /var/cloudstack/management/.ssh/id_rsa > > /usr/share/cloudstack-common/vms/systemvm.iso > > You should get e meaningful output if the key was injected or not > (perhaps > > the key inside doesn't differer, so the script will not inject it again) > > Proceed with what Pearl has suggested as the next steps. > > > > Best, > > > > On Thu, 3 Sep 2020 at 06:16, Pearl d'Silva <pearl.dsi...@shapeblue.com> > > wrote: > > > > > Hi Ammad, > > > > > > Could you try re-injecting the keys into the systemVM, as this MAY > happen > > > due to stale id_rsa keys. Run the injectkeys.sh to inject the existing > > > public ssh key into the authorized_keys file inside the systemvm.iso. > > > Execute: > > > > > > > > > /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh > > > /root/.ssh/id_rsa.pub /root/.ssh/id_rsa > > > /usr/share/cloudstack-common/vms/systemvm.iso' > > > > > > * Then replace the systemvm.iso on the hypervisor hosts. For each > > host: > > > > > > 1. Login and type command - xe host-param-clear param-name=tags > > > uuid=<uuid of the XS host> > > > host uuid maybe obtained using: xe host-list > > > > > > 2. 2. Restart MS and the iso should get propagated for all. > Check > > > timestamp. > > > 3. 3. Restart systemvms and they should get the iso inserted. > > > 4. > > > > > > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso#SystemVm.iso-Xen > > > > > > Thanks, > > > Pearl > > > > > > ________________________________ > > > From: Ammad Syed <syedamma...@gmail.com> > > > Sent: Wednesday, September 2, 2020 9:12 PM > > > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > > > Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error > > > > > > Guys, please advise how to troubleshoot this further ? How can i enable > > > trace level debugging ? any help in this will be highly appreciated? > > > > > > Ammad > > > > > > > > > pearl.dsi...@shapeblue.com > > > www.shapeblue.com > > > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > > > @shapeblue > > > > > > > > > > > > > On 27-Aug-2020, at 1:41 PM, Vivek Kumar <vivek.ku...@indiqus.com > > .invalid> > > > wrote: > > > > > > > > Pardon, the whole mail chain was not visible so replied accordingly. > > It > > > seems like you are doing SSH using root shell so it doesn’t matter. > > > > > > > > Vivek Kumar > > > > Manager - Cloud & DevOps > > > > IndiQus Technologies > > > > 24*7 O +91 11 4055 1411 | M +91 7503460090 > > > > www.indiqus.com<http://www.indiqus.com> <http://indiqus.com/> > > > > > > > > This message is intended only for the use of the individual or entity > > to > > > which it is addressed and may contain information that is confidential > > > and/or privileged. If you are not the intended recipient please delete > > the > > > original message and any copy of it from your computer system. You are > > > hereby notified that any dissemination, distribution or copying of this > > > communication is strictly prohibited unless proper authorization has > been > > > obtained for such action. If you have received this communication in > > error, > > > please notify the sender immediately. Although IndiQus attempts to > sweep > > > e-mail and attachments for viruses, it does not guarantee that both are > > > virus-free and accepts no liability for any damage sustained as a > result > > of > > > viruses. > > > > > > > >> On 27-Aug-2020, at 2:03 PM, Vivek Kumar <vivek.ku...@indiqus.com> > > > wrote: > > > >> > > > >> Hello Ammad, > > > >> > > > >> You haven’t defined the user while logging to the system VM through > > the > > > Link local IP that’s why you have got the error while logging to the > > system > > > VM. > > > >> > > > >>>> ssh -i /root/.ssh/id_rsa.cloud 169.254.0.121 -p 3922 > > > >> > > > >> Try instead below > > > >> > > > >>>> ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.0.121 > <mailto: > > > root@169.254.0.121> > > > >> > > > >> Vivek Kumar > > > >> Manager - Cloud & DevOps > > > >> IndiQus Technologies > > > >> 24*7 O +91 11 4055 1411 | M +91 7503460090 > > > >> www.indiqus.com<http://www.indiqus.com> <http://indiqus.com/> > > > >> > > > >> This message is intended only for the use of the individual or > entity > > > to which it is addressed and may contain information that is > confidential > > > and/or privileged. If you are not the intended recipient please delete > > the > > > original message and any copy of it from your computer system. You are > > > hereby notified that any dissemination, distribution or copying of this > > > communication is strictly prohibited unless proper authorization has > been > > > obtained for such action. If you have received this communication in > > error, > > > please notify the sender immediately. Although IndiQus attempts to > sweep > > > e-mail and attachments for viruses, it does not guarantee that both are > > > virus-free and accepts no liability for any damage sustained as a > result > > of > > > viruses. > > > >> > > > >>>> On 27-Aug-2020, at 1:59 PM, Ammad Syed <syedamma...@gmail.com > > > <mailto:syedamma...@gmail.com>> wrote: > > > >>> > > > >>> Guys, any help to troubleshoot this further would be highly > > > appreciated. > > > >>> > > > >>> > > > >>> Ammad Ali > > > >>>> On 18-Aug-2020, at 1:05 PM, Ammad Syed <syedamma...@gmail.com > > > <mailto:syedamma...@gmail.com>> wrote: > > > >>>> > > > >>>> > > > >>>> Hi, > > > >>>> > > > >>>> The systemVM is accessible with 169.x.x.x IP from xenserver host. > > But > > > login with private key is denied. I am testing this on my test > > environment. > > > >>>> > > > >>>> [root@xenserver ~]# ping 169.254.0.121 > > > >>>> PING 169.254.0.121 (169.254.0.121) 56(84) bytes of data. > > > >>>> 64 bytes from 169.254.0.121: icmp_seq=1 ttl=64 time=0.328 ms > > > >>>> 64 bytes from 169.254.0.121: icmp_seq=2 ttl=64 time=0.124 ms > > > >>>> 64 bytes from 169.254.0.121: icmp_seq=3 ttl=64 time=0.111 ms > > > >>>> 64 bytes from 169.254.0.121: icmp_seq=4 ttl=64 time=0.126 ms > > > >>>> ^C > > > >>>> --- 169.254.0.121 ping statistics --- > > > >>>> 4 packets transmitted, 4 received, 0% packet loss, time 2997ms > > > >>>> rtt min/avg/max/mdev = 0.111/0.172/0.328/0.090 ms > > > >>>> [root@xenserver ~]# > > > >>>> [root@xenserver ~]# > > > >>>> [root@xenserver ~]# ssh -i /root/.ssh/id_rsa.cloud 169.254.0.121 > -p > > > 3922 > > > >>>> Permission denied (publickey). > > > >>>> > > > >>>> I have checked by logging into the systemVM, the public key is not > > > found in /root/.ssh/authorized_keys. > > > >>>> > > > >>>> I have digged further, the systemvm.iso is same on the management > > > node and xenserver host. I have checked by mounting the ISO as well, > the > > > public key is there in authorized_keys in ISO. > > > >>>> > > > >>>> Restarting / recreating systemVM isn't fixed the issue. > > > >>>> > > > >>>> The systemVM is already running on HVM mode on xenserver host as > the > > > template OS type is Debian 8 64bit is selected. > > > >>>> > > > >>>> On my PRD upgrade, I have tested on one zone only. Currently I am > > > testing on my test ACS environment which have only one zone. > > > >>>> > > > >>>> Ammad Ali > > > >>>> > > > >>>>> On Mon, Aug 17, 2020 at 11:10 PM Andrija Panic < > > > andrija.pa...@gmail.com <mailto:andrija.pa...@gmail.com>> wrote: > > > >>>>> It was my tool (GUI based tool, that still doesn't return any > line > > > (???) > > > >>>>> but I see it. > > > >>>>> > > > >>>>> Problem is with 169.254.1.178 not being reachable over SSH > > > (2020-07-25 > > > >>>>> 01:45:52,097 DEBUG [c.c.h.x.r.CitrixResourceBase] > > > >>>>> (DirectAgent-57:ctx-ded7f7f9) (logid:a75971c2) Executing command > in > > > VR: > > > >>>>> /opt/cloud/bin/router_proxy.sh keystore-setup 169.254.1.178 > > > >>>>> /usr/local/cloud/systemvm/conf/agent.properties > > > >>>>> /usr/local/cloud/systemvm/conf/cloud.jks Xe8zzqZmUp9wmABH 365 > > > >>>>> /usr/local/cloud/systemvm/conf/cloud.csr) > > > >>>>> > > > >>>>> Can you try to ssh from that particular XenServer (where the SSVM > > is > > > >>>>> running) to the IP 169.xxxxx on port 3922 (ssh port) using the > > > private key > > > >>>>> - if you log in to the VM via it's console (root/password) - do > you > > > see > > > >>>>> with ifconfig the interface being started with 169..xxxx IP > > address? > > > I > > > >>>>> assume that destroying the SSVM and starting a new one doesn't > fix > > > the > > > >>>>> issue? > > > >>>>> > > > >>>>> You can also try changing the OS type of the systemVM template to > > > Other > > > >>>>> Linux 64bit, so it runs in HVM mode (instead of PV) - you'll need > > to > > > see ID > > > >>>>> for that OS type from the guest_os table, and set the guest_os_id > > > field in > > > >>>>> the vm_template table for your systemVM template. > > > >>>>> > > > >>>>> Does the issue occur in other zones as well? (assuming same > > > hypervisor type) > > > >>>>> > > > >>>>> > > > >>>>> On Mon, 17 Aug 2020 at 15:31, Ammad Syed <syedamma...@gmail.com > > > <mailto:syedamma...@gmail.com>> wrote: > > > >>>>> > > > >>>>>> I have rechecked the logs are there. Please review below lines. > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > > > > C:\Users\ammad\Downloads\management-server.log-4.13\management-server.log-4.13.1-errors > > > >>>>>> (27 hits) > > > >>>>>> Line 409969: 2020-07-25 01:45:52,320 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-58:ctx-9dc75334 > > > >>>>>> job-1207969/job-1208104 ctx-09e752ec) (logid:a75971c2) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: s-25023-VM > > > >>>>>> Line 437413: 2020-07-25 01:51:33,845 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-70:ctx-457ce4ee > > > >>>>>> job-1207969/job-1208129 ctx-195da69e) (logid:a75971c2) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: s-25024-VM > > > >>>>>> Line 549538: 2020-07-25 02:06:44,833 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-3:ctx-4634f7e0 > > > >>>>>> job-1208154/job-1208161 ctx-05d065dc) (logid:91fc037a) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-25027-VM > > > >>>>>> Line 663111: 2020-07-25 02:23:43,783 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-22:ctx-fe0b857b > > > >>>>>> job-1208154/job-1208222 ctx-ef088cd7) (logid:91fc037a) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-20771-VM > > > >>>>>> Line 667060: 2020-07-25 02:24:30,221 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-23:ctx-662cfe5d > > > >>>>>> job-1208154/job-1208229 ctx-e59878e9) (logid:91fc037a) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-24763-VM > > > >>>>>> Line 674269: 2020-07-25 02:25:47,345 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-24:ctx-b6529613 > > > >>>>>> job-1208154/job-1208233 ctx-7da81cab) (logid:91fc037a) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-20773-VM > > > >>>>>> Line 678169: 2020-07-25 02:26:20,355 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-31:ctx-7a062a66 > > > >>>>>> job-1208154/job-1208244 ctx-4f48d08a) (logid:91fc037a) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-25027-VM > > > >>>>>> Line 697113: 2020-07-25 02:30:09,669 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-40:ctx-3be9fddc > > > >>>>>> job-1208154/job-1208257 ctx-ed08fc43) (logid:91fc037a) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-20771-VM > > > >>>>>> Line 701161: 2020-07-25 02:30:48,127 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-41:ctx-4e3c666d > > > >>>>>> job-1208155/job-1208258 ctx-df740f75) (logid:9fa7dece) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: s-24142-VM > > > >>>>>> Line 701838: 2020-07-25 02:30:56,759 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-42:ctx-04ac8e71 > > > >>>>>> job-1208154/job-1208259 ctx-36c67b5e) (logid:91fc037a) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-24763-VM > > > >>>>>> Line 703865: 2020-07-25 02:31:25,123 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-46:ctx-24bdbe8f > > > >>>>>> job-1208154/job-1208266 ctx-fa500d3c) (logid:91fc037a) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-20773-VM > > > >>>>>> Line 712078: 2020-07-25 02:32:28,666 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-48:ctx-4d5db5c2 > > > >>>>>> job-1208155/job-1208269 ctx-b0f67c2e) (logid:9fa7dece) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: s-25033-VM > > > >>>>>> Line 715204: 2020-07-25 02:33:02,875 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-51:ctx-594da6d2 > > > >>>>>> job-1208155/job-1208273 ctx-ca115800) (logid:9fa7dece) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: s-25035-VM > > > >>>>>> Line 743140: 2020-07-25 02:38:52,257 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-59:ctx-a3f032da > > > >>>>>> job-1208155/job-1208294 ctx-626a2716) (logid:9fa7dece) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: s-25037-VM > > > >>>>>> Line 817646: 2020-07-25 02:47:07,127 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-5:ctx-ca073bd1 > > > >>>>>> job-1208297/job-1208305 ctx-6213af3b) (logid:1a2a782c) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: s-25038-VM > > > >>>>>> Line 847150: 2020-07-25 02:51:46,293 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-8:ctx-9db8c548 > > > >>>>>> job-1208296/job-1208312 ctx-83c570ff) (logid:436d1800) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-25040-VM > > > >>>>>> Line 903949: 2020-07-25 02:57:48,515 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-4:ctx-f35fd59a > > > >>>>>> job-1208320/job-1208333 ctx-f81179ac) (logid:05eacf0d) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-25041-VM > > > >>>>>> Line 957628: 2020-07-25 03:04:13,286 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-24:ctx-dc18956a > > > >>>>>> job-1208320/job-1208381 ctx-b53f5d7c) (logid:05eacf0d) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-20771-VM > > > >>>>>> Line 961606: 2020-07-25 03:05:00,522 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-28:ctx-83e48421 > > > >>>>>> job-1208320/job-1208392 ctx-e6767f04) (logid:05eacf0d) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-24763-VM > > > >>>>>> Line 966633: 2020-07-25 03:05:29,767 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-30:ctx-897e2a66 > > > >>>>>> job-1208320/job-1208397 ctx-39c396e1) (logid:05eacf0d) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-20773-VM > > > >>>>>> Line 968892: 2020-07-25 03:05:53,648 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-36:ctx-68e45b80 > > > >>>>>> job-1208320/job-1208405 ctx-1aaf96dd) (logid:05eacf0d) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-25041-VM > > > >>>>>> Line 1003036: 2020-07-25 03:13:33,582 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-45:ctx-0f552e0e > > > >>>>>> job-1208320/job-1208422 ctx-e8835fe9) (logid:05eacf0d) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-20771-VM > > > >>>>>> Line 1006496: 2020-07-25 03:14:09,787 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-46:ctx-08e5db24 > > > >>>>>> job-1208320/job-1208423 ctx-2cb636c2) (logid:05eacf0d) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-24763-VM > > > >>>>>> Line 1008408: 2020-07-25 03:14:39,584 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-47:ctx-8b9288a9 > > > >>>>>> job-1208320/job-1208424 ctx-c05d1544) (logid:05eacf0d) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-20773-VM > > > >>>>>> Line 1023914: 2020-07-25 03:15:58,139 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-48:ctx-4d54d717 > > > >>>>>> job-1208320/job-1208425 ctx-27e5397d) (logid:05eacf0d) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-25046-VM > > > >>>>>> Line 1089568: 2020-07-25 03:24:24,534 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-59:ctx-62ff8844 > > > >>>>>> job-1208320/job-1208449 ctx-d6a14f7f) (logid:05eacf0d) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: v-25048-VM > > > >>>>>> Line 1097605: 2020-07-25 03:25:23,179 ERROR > > > >>>>>> [c.c.v.VirtualMachineManagerImpl] > > (Work-Job-Executor-62:ctx-24407a9e > > > >>>>>> job-1208321/job-1208452 ctx-84f80ffa) (logid:42edff3a) Failed to > > > setup > > > >>>>>> keystore and generate CSR for system vm: s-25050-VM > > > >>>>>> > > > >>>>>> -Ammad Ali > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> On Mon, Aug 17, 2020 at 5:56 PM Andrija Panic < > > > andrija.pa...@gmail.com <mailto:andrija.pa...@gmail.com>> > > > >>>>>> wrote: > > > >>>>>> > > > >>>>>>> Ammad, no such log lines in the file you uploaded....? > > > >>>>>>> > > > >>>>>>> On Mon, 17 Aug 2020 at 09:02, Ammad Syed < > syedamma...@gmail.com > > > <mailto:syedamma...@gmail.com>> wrote: > > > >>>>>>> > > > >>>>>>>> Hi Andrija, > > > >>>>>>>> > > > >>>>>>>> The error occurred only one time. Can you please try to grep > > > "Failed to > > > >>>>>>>> setup keystore and generate CSR" you will see all the > systemVMs > > > that I > > > >>>>>>>> tried to create failed because of certificate generation > failed. > > > OR you > > > >>>>>>> can > > > >>>>>>>> grep job-1208321/job-1208452 this job as well. > > > >>>>>>>> > > > >>>>>>>> Also I mentioned above that same thing is happening in my test > > > >>>>>>> environment > > > >>>>>>>> that I created 4.11.1 then upgraded 4.11.3 > 4.13.1. > > > >>>>>>>> > > > >>>>>>>> Ammad Ali > > > >>>>>>>> > > > >>>>>>>> On Mon, Aug 17, 2020 at 11:17 AM Andrija Panic < > > > >>>>>> andrija.pa...@gmail.com <mailto:andrija.pa...@gmail.com>> > > > >>>>>>>> wrote: > > > >>>>>>>> > > > >>>>>>>>> 2020-07-25 03:23:51,093 WARN [c.c.v.SystemVmLoadScanner] > > > >>>>>>>>> (secstorage-1:ctx-50756e9b) (logid:3d7c25cb) Unexpected > > exception > > > >>>>>>> Failed > > > >>>>>>>> to > > > >>>>>>>>> fetch any free public IP address > > > >>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to > > fetch > > > any > > > >>>>>>> free > > > >>>>>>>>> public IP address > > > >>>>>>>>> > > > >>>>>>>>> ? > > > >>>>>>>>> > > > >>>>>>>>> On Sat, 15 Aug 2020 at 18:05, Ammad Syed < > > syedamma...@gmail.com > > > <mailto:syedamma...@gmail.com>> > > > >>>>>>> wrote: > > > >>>>>>>>> > > > >>>>>>>>>> I have checked further, the key is successfully injected in > > > >>>>>>>> systemvm.iso > > > >>>>>>>>> on > > > >>>>>>>>>> the management node and copied successfully to xenserver. I > > have > > > >>>>>>>> checked > > > >>>>>>>>> on > > > >>>>>>>>>> systemVM there is no public key found in authorized_keys of > > > root. > > > >>>>>>>>>> > > > >>>>>>>>>> How can I troubleshoot this further ? how can I enable trace > > > >>>>>> logging > > > >>>>>>>> for > > > >>>>>>>>>> management server to see if there is something problematic > > > >>>>>> happening. > > > >>>>>>>>>> > > > >>>>>>>>>> Ammad Ali > > > >>>>>>>>>> > > > >>>>>>>>>> On Sat, Aug 15, 2020 at 1:28 PM Ammad Syed < > > > syedamma...@gmail.com <mailto:syedamma...@gmail.com>> > > > >>>>>>>>> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>>> Hi, > > > >>>>>>>>>>> > > > >>>>>>>>>>> I have setup my test environment, here is what I did: > > > >>>>>>>>>>> > > > >>>>>>>>>>> - Installed ACS 4.11.1 and added xenserver 7.0 host in it. > > > >>>>>>> SystemVMs > > > >>>>>>>>> are > > > >>>>>>>>>>> up in the zone and agents are up. > > > >>>>>>>>>>> - Then upgraded the system to 4.11.3. Recreated systemVMs, > > > agent > > > >>>>>> is > > > >>>>>>>> up > > > >>>>>>>>>> and > > > >>>>>>>>>>> systemVM are up with 4.11.3 systemVM version. > > > >>>>>>>>>>> - Then upgraded the system to 4.13.1, recreated systemVMs > are > > > >>>>>>> running > > > >>>>>>>>> but > > > >>>>>>>>>>> agents are not up. > > > >>>>>>>>>>> > > > >>>>>>>>>>> I have checked md5sum of systemvm.iso on xenserver and > > > management > > > >>>>>>>>> server, > > > >>>>>>>>>>> both are same. > > > >>>>>>>>>>> > > > >>>>>>>>>>> [root@xenserver iso]# md5sum > > > >>>>>>>> /opt/xensource/packages/iso/systemvm.iso > > > >>>>>>>>>>> baba18f156395da3a5d8208727d8f421 > > > >>>>>>>>>> /opt/xensource/packages/iso/systemvm.iso > > > >>>>>>>>>>> > > > >>>>>>>>>>> [root@cloudstack-upgrade vms]# md5sum > > > >>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso > > > >>>>>>>>>>> baba18f156395da3a5d8208727d8f421 > > > >>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso > > > >>>>>>>>>>> > > > >>>>>>>>>>> Also the private key on xenserver root and management > server > > > are > > > >>>>>>>> same. > > > >>>>>>>>>>> > > > >>>>>>>>>>> management server > > > >>>>>>>>>>> > /usr/share/cloudstack-common/scripts/vm/systemvm/id_rsa.cloud > > > >>>>>>>>>>> > > > >>>>>>>>>>> xenserver path > > > >>>>>>>>>>> /root/.ssh/id_rsa.cloud > > > >>>>>>>>>>> > > > >>>>>>>>>>> - Ammad Ali > > > >>>>>>>>>>> > > > >>>>>>>>>>> On Thu, Aug 13, 2020 at 11:27 AM Ammad Syed < > > > >>>>>> syedamma...@gmail.com <mailto:syedamma...@gmail.com> > > > >>>>>>>> > > > >>>>>>>>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>>> Here is the link for download management logs. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > > > > https://drive.google.com/file/d/1l6HDPguGUNaOxsc7VSaj7eaP0F2UOFjA/view?usp=sharing > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> On Thu, Aug 13, 2020 at 11:22 AM Ammad Syed < > > > >>>>>>> syedamma...@gmail.com> > > > >>>>>>>>>>>> wrote: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> I have reverted the version back to 4.11.3. But I have > > saved > > > >>>>>> logs > > > >>>>>>>>>>>>> starting from upgrade. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> I think the key has been copied successfully in system vm > > > iso. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> 2020-07-25 00:34:17,214 INFO > > [c.c.s.ConfigurationServerImpl] > > > >>>>>>>>>>>>> (main:null) (logid:) Going to update systemvm iso with > > > >>>>>> generated > > > >>>>>>>>>> keypairs > > > >>>>>>>>>>>>> if needed > > > >>>>>>>>>>>>> 2020-07-25 00:34:17,214 INFO > > [c.c.s.ConfigurationServerImpl] > > > >>>>>>>>>>>>> (main:null) (logid:) Trying to inject public and private > > keys > > > >>>>>>> into > > > >>>>>>>>>> systemvm > > > >>>>>>>>>>>>> iso > > > >>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] > (main:null) > > > >>>>>>> (logid:) > > > >>>>>>>>>>>>> Looking for vms/systemvm.iso in the classpath > > > >>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] > (main:null) > > > >>>>>>> (logid:) > > > >>>>>>>>>>>>> System resource: > > > >>>>>>> file:/usr/share/cloudstack-common/vms/systemvm.iso > > > >>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] > (main:null) > > > >>>>>>> (logid:) > > > >>>>>>>>>>>>> Absolute path = > > > /usr/share/cloudstack-common/vms/systemvm.iso > > > >>>>>>>>>>>>> 2020-07-25 00:34:17,218 DEBUG > > [c.c.s.ConfigurationServerImpl] > > > >>>>>>>>>>>>> (main:null) (logid:) Executing: /bin/bash > > > >>>>>>>>>>>>> > > > /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh > > > >>>>>>>>>>>>> /var/cloudstack/management/.ssh/id_rsa.pub > > > >>>>>>>>>>>>> /var/cloudstack/management/.ssh/id_rsa > > > >>>>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso > > > >>>>>>>>>>>>> 2020-07-25 00:34:17,636 INFO > > [c.c.s.ConfigurationServerImpl] > > > >>>>>>>>>>>>> (main:null) (logid:) Injected public and private keys > into > > > >>>>>>> systemvm > > > >>>>>>>>> iso > > > >>>>>>>>>>>>> with result : null > > > >>>>>>>>>>>>> 2020-07-25 00:34:50,613 DEBUG > > [c.c.h.x.r.CitrixResourceBase] > > > >>>>>>>>>>>>> (DirectAgent-1:ctx-d3dc4cf2) (logid:1d22396d) Copying > > > >>>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > > > > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso > > > >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.22 with > > permission > > > >>>>>>> 0644 > > > >>>>>>>>>>>>> 2020-07-25 00:34:52,566 DEBUG > > [c.c.h.x.r.CitrixResourceBase] > > > >>>>>>>>>>>>> (DirectAgent-2:ctx-2537b610) (logid:29c67b7a) Copying > > > >>>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > > > > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso > > > >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.5 with > > permission > > > >>>>>> 0644 > > > >>>>>>>>>>>>> 2020-07-25 00:34:53,170 DEBUG > > [c.c.h.x.r.CitrixResourceBase] > > > >>>>>>>>>>>>> (DirectAgent-3:ctx-168ac27d) (logid:a7862c4b) Copying > > > >>>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > > > > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso > > > >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.9 with > > permission > > > >>>>>> 0644 > > > >>>>>>>>>>>>> 2020-07-25 00:34:54,621 DEBUG > > [c.c.h.x.r.CitrixResourceBase] > > > >>>>>>>>>>>>> (DirectAgent-6:ctx-f640fe55) (logid:0a62d4cf) Copying > > > >>>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > > > > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso > > > >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.5 with > > permission > > > >>>>>> 0644 > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> I have attached complete management logs. The start of > logs > > > is > > > >>>>>>> the > > > >>>>>>>>>> start > > > >>>>>>>>>>>>> of management server after upgrade of management server > > from > > > >>>>>>> 4.11.3 > > > >>>>>>>>> to > > > >>>>>>>>>>>>> 4.13.1. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Ammad Ali > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> On Thu, Aug 13, 2020 at 1:35 AM Andrija Panic < > > > >>>>>>>>> andrija.pa...@gmail.com > > > >>>>>>>>>>> > > > >>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Do you get an error while trying to inject ssh key into > > the > > > >>>>>>>>>> systemvm.iso > > > >>>>>>>>>>>>>> (mgmt logs) , or can you confirm that the systemvm.iso > on > > XS > > > >>>>>>> host > > > >>>>>>>>> and > > > >>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>> one on the mgmt server are identical, md5sum them (i.e. > > has > > > >>>>>> the > > > >>>>>>>> iso > > > >>>>>>>>>> been > > > >>>>>>>>>>>>>> copied over to the XS successfully) - this might explain > > not > > > >>>>>>> being > > > >>>>>>>>>> able > > > >>>>>>>>>>>>>> to > > > >>>>>>>>>>>>>> login via ssh with your private key. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Best, > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> On Wed, 12 Aug 2020, 09:08 Ammad Syed, < > > > syedamma...@gmail.com > > > >>>>>>> > > > >>>>>>>>> wrote: > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Yes this is exactly the same issue that i faced. > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Sent from my iPhone > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> On 12-Aug-2020, at 8:35 AM, Eric Lee Green < > > > >>>>>>>>>>>>>> eric.lee.gr...@gmail.com> > > > >>>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Correct, 4.11.3 template is used for 4.11.3, 4.12, > and > > > >>>>>>> 4.13. > > > >>>>>>>>> 4.14 > > > >>>>>>>>>>>>>> moves > > > >>>>>>>>>>>>>>> to the 4.14.0 template. > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> There seems to be something odd happening key-wise > > > >>>>>> sometimes > > > >>>>>>>>> with > > > >>>>>>>>>>>>>>> upgrades from 4.11.3 to 4.13.1 or 4.14.0. I managed > an > > > >>>>>>> upgrade > > > >>>>>>>>>> from > > > >>>>>>>>>>>>>>> 4.11.3 to 4.13.1 that *almost* worked, but the > secondary > > > >>>>>>> storage > > > >>>>>>>>> VM > > > >>>>>>>>>>>>>>> wouldn't work and thus I couldn't spawn new virtual > > > >>>>>> machines. > > > >>>>>>>> Same > > > >>>>>>>>>>>>>> symptom > > > >>>>>>>>>>>>>>> -- key error when the agent tried to ssh into it. And > > > >>>>>> deleting > > > >>>>>>>> it > > > >>>>>>>>>> and > > > >>>>>>>>>>>>>>> making it respawn didn't help. Then I tried 4.11.3 to > > > 4.14.0 > > > >>>>>>> and > > > >>>>>>>>>>>>>> *all* the > > > >>>>>>>>>>>>>>> VM's failed at that point (of course, that was with the > > new > > > >>>>>>>>>> template). > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Right now I'm back at 4.11.3 until this can be figured > > > >>>>>> out. > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> On 8/11/2020 5:53 AM, Ammad Syed wrote: > > > >>>>>>>>>>>>>>>>> Hi, > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> I think 4.12 and 4.13 uses same systemVM template i.e > > > >>>>>>> 4.11.3 > > > >>>>>>>>>>>>>> version, > > > >>>>>>>>>>>>>>> which > > > >>>>>>>>>>>>>>>>> I already have registered. Currently I am running > > 4.11.3 > > > >>>>>>>>> version > > > >>>>>>>>>>>>>> of ACS. > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> MariaDB [cloud]> SELECT > id,name,type,cross_zones,state > > > >>>>>> FROM > > > >>>>>>>>>>>>>>>>> cloud.vm_template WHERE name like > > '%systemvm-xenserver%' > > > >>>>>>> AND > > > >>>>>>>>>>>>>> removed IS > > > >>>>>>>>>>>>>>>>> NULL; > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>> > > > >>>>>> > > > > +------+-----------------------------+--------+-------------+----------+ > > > >>>>>>>>>>>>>>>>> | id | name | type | > > > >>>>>>> cross_zones | > > > >>>>>>>>>>>>>> state | > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>> > > > >>>>>> > > > > +------+-----------------------------+--------+-------------+----------+ > > > >>>>>>>>>>>>>>>>> | 337 | systemvm-xenserver-3.0.0 | SYSTEM | > > > >>>>>>> 0 | > > > >>>>>>>>>>>>>> Inactive | > > > >>>>>>>>>>>>>>>>> | 418 | systemvm-xenserver-4.2 | SYSTEM | > > > >>>>>>> 0 | > > > >>>>>>>>>>>>>> Active | > > > >>>>>>>>>>>>>>>>> | 472 | systemvm-xenserver-4.3 | USER | > > > >>>>>>> 1 | > > > >>>>>>>>>>>>>> Inactive | > > > >>>>>>>>>>>>>>>>> | 473 | systemvm-xenserver-4.3 | USER | > > > >>>>>>> 1 | > > > >>>>>>>>>>>>>> Inactive | > > > >>>>>>>>>>>>>>>>> | 474 | systemvm-xenserver-4.3 | USER | > > > >>>>>>> 1 | > > > >>>>>>>>>>>>>> Inactive | > > > >>>>>>>>>>>>>>>>> | 475 | systemvm-xenserver-4.3 | USER | > > > >>>>>>> 1 | > > > >>>>>>>>>>>>>> Inactive | > > > >>>>>>>>>>>>>>>>> | 476 | systemvm-xenserver-4.3 | USER | > > > >>>>>>> 0 | > > > >>>>>>>>>>>>>> Inactive | > > > >>>>>>>>>>>>>>>>> | 479 | systemvm-xenserver-4.3-2 | USER | > > > >>>>>>> 1 | > > > >>>>>>>>>>>>>> Inactive | > > > >>>>>>>>>>>>>>>>> | 480 | systemvm-xenserver-4.3 | SYSTEM | > > > >>>>>>> 0 | > > > >>>>>>>>>>>>>> Active | > > > >>>>>>>>>>>>>>>>> | 549 | systemvm-xenserver-4.5.1 | USER | > > > >>>>>>> 0 | > > > >>>>>>>>>>>>>> Active | > > > >>>>>>>>>>>>>>>>> | 550 | systemvm-xenserver-4.5.1 | SYSTEM | > > > >>>>>>> 0 | > > > >>>>>>>>>>>>>> Active | > > > >>>>>>>>>>>>>>>>> | 651 | systemvm-xenserver-4.7.0 | USER | > > > >>>>>>> 0 | > > > >>>>>>>>>>>>>> Inactive | > > > >>>>>>>>>>>>>>>>> | 652 | systemvm-xenserver-4.7.0 | USER | > > > >>>>>>> 0 | > > > >>>>>>>>>>>>>> Inactive | > > > >>>>>>>>>>>>>>>>> | 653 | systemvm-xenserver-4.7.0 | SYSTEM | > > > >>>>>>> 0 | > > > >>>>>>>>>>>>>> Inactive | > > > >>>>>>>>>>>>>>>>> | 737 | systemvm-xenserver-4.9.2 | SYSTEM | > > > >>>>>>> 1 | > > > >>>>>>>>>>>>>> Inactive | > > > >>>>>>>>>>>>>>>>> | 739 | systemvm-xenserver-4.9.2-sb | SYSTEM | > > > >>>>>>> 1 | > > > >>>>>>>>>>>>>> Active | > > > >>>>>>>>>>>>>>>>> | 1245 | systemvm-xenserver-4.11.1 | SYSTEM | > > > >>>>>>> 1 | > > > >>>>>>>>>>>>>> Active | > > > >>>>>>>>>>>>>>>>> | 1584 | systemvm-xenserver-4.11.2 | SYSTEM | > > > >>>>>>> 1 | > > > >>>>>>>>>>>>>> Active | > > > >>>>>>>>>>>>>>>>> | 1677 | systemvm-xenserver-4.11.3 | SYSTEM | > > > >>>>>>> 1 | > > > >>>>>>>>>>>>>> Active | > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>> > > > >>>>>> > > > > +------+-----------------------------+--------+-------------+----------+ > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> - Ammad > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> On Tue, Aug 11, 2020 at 5:17 PM Pierre-Luc Dion < > > > >>>>>>>>>>>>>> pdion...@apache.org> > > > >>>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>>> db. > > > >>>>>>>>>>>>>>>>>> Hi Syed, > > > >>>>>>>>>>>>>>>>>> From 4.12, the systemvm template had to be upgraded > > > >>>>>>> because > > > >>>>>>>> of > > > >>>>>>>>>> OS > > > >>>>>>>>>>>>>>> change in > > > >>>>>>>>>>>>>>>>>> the template, moved to a latest version of Debian. > > > >>>>>> Because > > > >>>>>>>> of > > > >>>>>>>>>>>>>> that, > > > >>>>>>>>>>>>>>> some VR > > > >>>>>>>>>>>>>>>>>> scripts have changed and make obsolete older version > > of > > > >>>>>>> VRs, > > > >>>>>>>>> so > > > >>>>>>>>>>>>>> you > > > >>>>>>>>>>>>>>> will > > > >>>>>>>>>>>>>>>>>> most likely have to register an updated systemvm > > > >>>>>> templates > > > >>>>>>>> and > > > >>>>>>>>>>>>>> upgrade > > > >>>>>>>>>>>>>>> your > > > >>>>>>>>>>>>>>>>>> system VMs and VRs. > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> Regards, > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> On Tue, Aug 11, 2020 at 6:24 AM Ammad Syed < > > > >>>>>>>>>>>>>> syedamma...@gmail.com> > > > >>>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> Hi Guys, > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> I was previously on 4.9.3 cloudstack and upgraded > to > > > >>>>>>> 4.11.1 > > > >>>>>>>>>> then > > > >>>>>>>>>>>>>>> 4.11.3. > > > >>>>>>>>>>>>>>>>>>> The version 4.11.3 is working fine since six > months. > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> Now I have tried to upgrade my system from 4.11.3 > to > > > >>>>>>>> 4.13.1. > > > >>>>>>>>>> The > > > >>>>>>>>>>>>>>> upgrade > > > >>>>>>>>>>>>>>>>>>> goes successful. I didn't uploaded any system VM > > > >>>>>>> template. > > > >>>>>>>>>>>>>> However the > > > >>>>>>>>>>>>>>>>>>> problem occured when I recreated my systemVM of > POD, > > > >>>>>> the > > > >>>>>>> VM > > > >>>>>>>>>>>>>> recreated > > > >>>>>>>>>>>>>>> and > > > >>>>>>>>>>>>>>>>>>> its state was running but agent state was not > getting > > > >>>>>> up, > > > >>>>>>>> its > > > >>>>>>>>>>>>>> showing > > > >>>>>>>>>>>>>>>>>> blank > > > >>>>>>>>>>>>>>>>>>> in column. > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> Digging further via job logs, the job is failed > with > > > >>>>>>> error > > > >>>>>>>>> that > > > >>>>>>>>>>>>>>> unable to > > > >>>>>>>>>>>>>>>>>>> execute command via ssh. Below are the logs. > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,126 ERROR [c.c.u.s.SshHelper] > > > >>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) SSH > > > >>>>>>>> execution > > > >>>>>>>>>> of > > > >>>>>>>>>>>>>>> command > > > >>>>>>>>>>>>>>>>>>> /opt/cloud/bin/router_proxy.sh keystore-s > > > >>>>>>>>>>>>>>>>>>> etup 169.254.2.199 > > > >>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/agent.properties > > > >>>>>>>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/cloud.jks > > > >>>>>> TJaQYChYBwKh7Cx9 > > > >>>>>>>> 365 > > > >>>>>>>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/clou > > > >>>>>>>>>>>>>>>>>>> d.csr has an error status code in return. Result > > > >>>>>> output: > > > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG > > > >>>>>>> [c.c.a.m.DirectAgentAttache] > > > >>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq > > > >>>>>>>>>>>>>>>>>>> 906-3195585410596077730: Response Received: > > > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request] > > > >>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq > > > >>>>>>>>>>>>>>>>>>> 906-3195585410596077730: Processing: { Ans: , > > MgmtId: > > > >>>>>>>>>> 779271079 > > > >>>>>>>>>>>>>>>>>>> 43497, via: 906(xen-21-10-a3-khi02), Ver: v1, > Flags: > > > >>>>>> 10, > > > >>>>>>>>>>>>>>>>>>> [{"org.apache.cloudstack.ca > > > >>>>>>>>>>>>>>>>>>> .SetupKeystoreAnswer":{"result":false,"wait":0}}] > > > >>>>>>>>>>>>>>>>>>> } > > > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request] > > > >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d > > > >>>>>>> job-1208155/job-1208258 > > > >>>>>>>>>>>>>>> ctx-df740f75) > > > >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Seq 906-319558541059607773 > > > >>>>>>>>>>>>>>>>>>> 0: Received: { Ans: , MgmtId: 77927107943497, via: > > > >>>>>>>>>>>>>>>>>>> 906(xen-21-10-a3-khi02), Ver: v1, Flags: 10, { > > > >>>>>>>>>>>>>> SetupKeystoreAnswer } } > > > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 ERROR > > > >>>>>>>>> [c.c.v.VirtualMachineManagerImpl] > > > >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d > > > >>>>>>> job-1208155/job-1208258 > > > >>>>>>>>>>>>>>> ctx-df740f75) > > > >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Failed to setup keystore and > > generate > > > >>>>>>> CSR > > > >>>>>>>>> for > > > >>>>>>>>>>>>>> system > > > >>>>>>>>>>>>>>> vm: > > > >>>>>>>>>>>>>>>>>>> s-24142-VM > > > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG > > > >>>>>>> [c.c.v.VmWorkJobHandlerProxy] > > > >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d > > > >>>>>>> job-1208155/job-1208258 > > > >>>>>>>>>>>>>>> ctx-df740f75) > > > >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Done executing VM work job: > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > > > > com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":24142,"handlerName":"VirtualMachineManagerImpl"} > > > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,128 DEBUG > > > >>>>>>>>> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d > > > >>>>>>> job-1208155/job-1208258 > > > >>>>>>>>>>>>>>> ctx-df740f75) > > > >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Complete async job-1208258, > > jobStatus: > > > >>>>>>>>>>>>>> SUCCEEDED, > > > >>>>>>>>>>>>>>>>>>> resultCode: 0, result: null > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> I tried to dig it further, I was unable to login > > > >>>>>> systemVM > > > >>>>>>>> via > > > >>>>>>>>>>>>>> ssh from > > > >>>>>>>>>>>>>>>>>>> xenserver host with key /root/.ssh/id_rsa.cloud > > placed. > > > >>>>>>>> Look > > > >>>>>>>>>> like > > > >>>>>>>>>>>>>>> private > > > >>>>>>>>>>>>>>>>>>> key issue. However I am able to login on my old > > > >>>>>>> systemVMs ( > > > >>>>>>>>> i.e > > > >>>>>>>>>>>>>>> created > > > >>>>>>>>>>>>>>>>>> on > > > >>>>>>>>>>>>>>>>>>> ACS 4.11.3) > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> Also I have SSL certificate enabled for console > proxy > > > >>>>>> on > > > >>>>>>> my > > > >>>>>>>>> ACS > > > >>>>>>>>>>>>>> 4.11.3 > > > >>>>>>>>>>>>>>>>>> and > > > >>>>>>>>>>>>>>>>>>> I am using only xenserver 7.0 hosts. > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> I tried to disable SSL on secstorage and console > > proxy > > > >>>>>>> from > > > >>>>>>>>>>>>>> global > > > >>>>>>>>>>>>>>>>>>> settings, but still didn't worked. > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> I had a fresh installation of ACS 4.13.1 with > > xenserver > > > >>>>>>>> 7.0, > > > >>>>>>>>>>>>>> systemVMs > > > >>>>>>>>>>>>>>>>>> are > > > >>>>>>>>>>>>>>>>>>> working fine in it. > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> Please advise. > > > >>>>>>>>>>>>>>>>>>> -- > > > >>>>>>>>>>>>>>>>>>> Regards, > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> Syed Ammad Ali > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> -- > > > >>>>>>>>>>>>> Regards, > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Syed Ammad Ali > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> -- > > > >>>>>>>>>>>> Regards, > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Syed Ammad Ali > > > >>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> -- > > > >>>>>>>>>>> Regards, > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> Syed Ammad Ali > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> -- > > > >>>>>>>>>> Regards, > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> Syed Ammad Ali > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> -- > > > >>>>>>>>> > > > >>>>>>>>> Andrija Panić > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> -- > > > >>>>>>>> Regards, > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> Syed Ammad Ali > > > >>>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> -- > > > >>>>>>> > > > >>>>>>> Andrija Panić > > > >>>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> -- > > > >>>>>> Regards, > > > >>>>>> > > > >>>>>> > > > >>>>>> Syed Ammad Ali > > > >>>>>> > > > >>>>> > > > >>>>> > > > >>>>> -- > > > >>>>> > > > >>>>> Andrija Panić > > > >>>> > > > >>>> > > > >>>> -- > > > >>>> Regards, > > > >>>> > > > >>>> > > > >>>> Syed Ammad Ali > > > >> > > > > > > > > > > > > > -- > > > > Andrija Panić > > > > > -- > Regards, > > > Syed Ammad Ali > -- Andrija Panić
