It was my tool (GUI based tool, that still doesn't return any line (???) but I see it.
Problem is with 169.254.1.178 not being reachable over SSH (2020-07-25 01:45:52,097 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-57:ctx-ded7f7f9) (logid:a75971c2) Executing command in VR: /opt/cloud/bin/router_proxy.sh keystore-setup 169.254.1.178 /usr/local/cloud/systemvm/conf/agent.properties /usr/local/cloud/systemvm/conf/cloud.jks Xe8zzqZmUp9wmABH 365 /usr/local/cloud/systemvm/conf/cloud.csr) Can you try to ssh from that particular XenServer (where the SSVM is running) to the IP 169.xxxxx on port 3922 (ssh port) using the private key - if you log in to the VM via it's console (root/password) - do you see with ifconfig the interface being started with 169..xxxx IP address? I assume that destroying the SSVM and starting a new one doesn't fix the issue? You can also try changing the OS type of the systemVM template to Other Linux 64bit, so it runs in HVM mode (instead of PV) - you'll need to see ID for that OS type from the guest_os table, and set the guest_os_id field in the vm_template table for your systemVM template. Does the issue occur in other zones as well? (assuming same hypervisor type) On Mon, 17 Aug 2020 at 15:31, Ammad Syed <syedamma...@gmail.com> wrote: > I have rechecked the logs are there. Please review below lines. > > > > C:\Users\ammad\Downloads\management-server.log-4.13\management-server.log-4.13.1-errors > (27 hits) > Line 409969: 2020-07-25 01:45:52,320 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-58:ctx-9dc75334 > job-1207969/job-1208104 ctx-09e752ec) (logid:a75971c2) Failed to setup > keystore and generate CSR for system vm: s-25023-VM > Line 437413: 2020-07-25 01:51:33,845 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-70:ctx-457ce4ee > job-1207969/job-1208129 ctx-195da69e) (logid:a75971c2) Failed to setup > keystore and generate CSR for system vm: s-25024-VM > Line 549538: 2020-07-25 02:06:44,833 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-3:ctx-4634f7e0 > job-1208154/job-1208161 ctx-05d065dc) (logid:91fc037a) Failed to setup > keystore and generate CSR for system vm: v-25027-VM > Line 663111: 2020-07-25 02:23:43,783 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-22:ctx-fe0b857b > job-1208154/job-1208222 ctx-ef088cd7) (logid:91fc037a) Failed to setup > keystore and generate CSR for system vm: v-20771-VM > Line 667060: 2020-07-25 02:24:30,221 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-23:ctx-662cfe5d > job-1208154/job-1208229 ctx-e59878e9) (logid:91fc037a) Failed to setup > keystore and generate CSR for system vm: v-24763-VM > Line 674269: 2020-07-25 02:25:47,345 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-24:ctx-b6529613 > job-1208154/job-1208233 ctx-7da81cab) (logid:91fc037a) Failed to setup > keystore and generate CSR for system vm: v-20773-VM > Line 678169: 2020-07-25 02:26:20,355 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-31:ctx-7a062a66 > job-1208154/job-1208244 ctx-4f48d08a) (logid:91fc037a) Failed to setup > keystore and generate CSR for system vm: v-25027-VM > Line 697113: 2020-07-25 02:30:09,669 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-40:ctx-3be9fddc > job-1208154/job-1208257 ctx-ed08fc43) (logid:91fc037a) Failed to setup > keystore and generate CSR for system vm: v-20771-VM > Line 701161: 2020-07-25 02:30:48,127 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-41:ctx-4e3c666d > job-1208155/job-1208258 ctx-df740f75) (logid:9fa7dece) Failed to setup > keystore and generate CSR for system vm: s-24142-VM > Line 701838: 2020-07-25 02:30:56,759 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-42:ctx-04ac8e71 > job-1208154/job-1208259 ctx-36c67b5e) (logid:91fc037a) Failed to setup > keystore and generate CSR for system vm: v-24763-VM > Line 703865: 2020-07-25 02:31:25,123 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-46:ctx-24bdbe8f > job-1208154/job-1208266 ctx-fa500d3c) (logid:91fc037a) Failed to setup > keystore and generate CSR for system vm: v-20773-VM > Line 712078: 2020-07-25 02:32:28,666 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-4d5db5c2 > job-1208155/job-1208269 ctx-b0f67c2e) (logid:9fa7dece) Failed to setup > keystore and generate CSR for system vm: s-25033-VM > Line 715204: 2020-07-25 02:33:02,875 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-51:ctx-594da6d2 > job-1208155/job-1208273 ctx-ca115800) (logid:9fa7dece) Failed to setup > keystore and generate CSR for system vm: s-25035-VM > Line 743140: 2020-07-25 02:38:52,257 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-59:ctx-a3f032da > job-1208155/job-1208294 ctx-626a2716) (logid:9fa7dece) Failed to setup > keystore and generate CSR for system vm: s-25037-VM > Line 817646: 2020-07-25 02:47:07,127 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-5:ctx-ca073bd1 > job-1208297/job-1208305 ctx-6213af3b) (logid:1a2a782c) Failed to setup > keystore and generate CSR for system vm: s-25038-VM > Line 847150: 2020-07-25 02:51:46,293 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-8:ctx-9db8c548 > job-1208296/job-1208312 ctx-83c570ff) (logid:436d1800) Failed to setup > keystore and generate CSR for system vm: v-25040-VM > Line 903949: 2020-07-25 02:57:48,515 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-4:ctx-f35fd59a > job-1208320/job-1208333 ctx-f81179ac) (logid:05eacf0d) Failed to setup > keystore and generate CSR for system vm: v-25041-VM > Line 957628: 2020-07-25 03:04:13,286 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-24:ctx-dc18956a > job-1208320/job-1208381 ctx-b53f5d7c) (logid:05eacf0d) Failed to setup > keystore and generate CSR for system vm: v-20771-VM > Line 961606: 2020-07-25 03:05:00,522 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-28:ctx-83e48421 > job-1208320/job-1208392 ctx-e6767f04) (logid:05eacf0d) Failed to setup > keystore and generate CSR for system vm: v-24763-VM > Line 966633: 2020-07-25 03:05:29,767 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-30:ctx-897e2a66 > job-1208320/job-1208397 ctx-39c396e1) (logid:05eacf0d) Failed to setup > keystore and generate CSR for system vm: v-20773-VM > Line 968892: 2020-07-25 03:05:53,648 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-36:ctx-68e45b80 > job-1208320/job-1208405 ctx-1aaf96dd) (logid:05eacf0d) Failed to setup > keystore and generate CSR for system vm: v-25041-VM > Line 1003036: 2020-07-25 03:13:33,582 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-45:ctx-0f552e0e > job-1208320/job-1208422 ctx-e8835fe9) (logid:05eacf0d) Failed to setup > keystore and generate CSR for system vm: v-20771-VM > Line 1006496: 2020-07-25 03:14:09,787 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-46:ctx-08e5db24 > job-1208320/job-1208423 ctx-2cb636c2) (logid:05eacf0d) Failed to setup > keystore and generate CSR for system vm: v-24763-VM > Line 1008408: 2020-07-25 03:14:39,584 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-47:ctx-8b9288a9 > job-1208320/job-1208424 ctx-c05d1544) (logid:05eacf0d) Failed to setup > keystore and generate CSR for system vm: v-20773-VM > Line 1023914: 2020-07-25 03:15:58,139 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-4d54d717 > job-1208320/job-1208425 ctx-27e5397d) (logid:05eacf0d) Failed to setup > keystore and generate CSR for system vm: v-25046-VM > Line 1089568: 2020-07-25 03:24:24,534 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-59:ctx-62ff8844 > job-1208320/job-1208449 ctx-d6a14f7f) (logid:05eacf0d) Failed to setup > keystore and generate CSR for system vm: v-25048-VM > Line 1097605: 2020-07-25 03:25:23,179 ERROR > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-62:ctx-24407a9e > job-1208321/job-1208452 ctx-84f80ffa) (logid:42edff3a) Failed to setup > keystore and generate CSR for system vm: s-25050-VM > > -Ammad Ali > > > > On Mon, Aug 17, 2020 at 5:56 PM Andrija Panic <andrija.pa...@gmail.com> > wrote: > > > Ammad, no such log lines in the file you uploaded....? > > > > On Mon, 17 Aug 2020 at 09:02, Ammad Syed <syedamma...@gmail.com> wrote: > > > > > Hi Andrija, > > > > > > The error occurred only one time. Can you please try to grep "Failed to > > > setup keystore and generate CSR" you will see all the systemVMs that I > > > tried to create failed because of certificate generation failed. OR you > > can > > > grep job-1208321/job-1208452 this job as well. > > > > > > Also I mentioned above that same thing is happening in my test > > environment > > > that I created 4.11.1 then upgraded 4.11.3 > 4.13.1. > > > > > > Ammad Ali > > > > > > On Mon, Aug 17, 2020 at 11:17 AM Andrija Panic < > andrija.pa...@gmail.com> > > > wrote: > > > > > > > 2020-07-25 03:23:51,093 WARN [c.c.v.SystemVmLoadScanner] > > > > (secstorage-1:ctx-50756e9b) (logid:3d7c25cb) Unexpected exception > > Failed > > > to > > > > fetch any free public IP address > > > > com.cloud.utils.exception.CloudRuntimeException: Failed to fetch any > > free > > > > public IP address > > > > > > > > ? > > > > > > > > On Sat, 15 Aug 2020 at 18:05, Ammad Syed <syedamma...@gmail.com> > > wrote: > > > > > > > > > I have checked further, the key is successfully injected in > > > systemvm.iso > > > > on > > > > > the management node and copied successfully to xenserver. I have > > > checked > > > > on > > > > > systemVM there is no public key found in authorized_keys of root. > > > > > > > > > > How can I troubleshoot this further ? how can I enable trace > logging > > > for > > > > > management server to see if there is something problematic > happening. > > > > > > > > > > Ammad Ali > > > > > > > > > > On Sat, Aug 15, 2020 at 1:28 PM Ammad Syed <syedamma...@gmail.com> > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I have setup my test environment, here is what I did: > > > > > > > > > > > > - Installed ACS 4.11.1 and added xenserver 7.0 host in it. > > SystemVMs > > > > are > > > > > > up in the zone and agents are up. > > > > > > - Then upgraded the system to 4.11.3. Recreated systemVMs, agent > is > > > up > > > > > and > > > > > > systemVM are up with 4.11.3 systemVM version. > > > > > > - Then upgraded the system to 4.13.1, recreated systemVMs are > > running > > > > but > > > > > > agents are not up. > > > > > > > > > > > > I have checked md5sum of systemvm.iso on xenserver and management > > > > server, > > > > > > both are same. > > > > > > > > > > > > [root@xenserver iso]# md5sum > > > /opt/xensource/packages/iso/systemvm.iso > > > > > > baba18f156395da3a5d8208727d8f421 > > > > > /opt/xensource/packages/iso/systemvm.iso > > > > > > > > > > > > [root@cloudstack-upgrade vms]# md5sum > > > > > > /usr/share/cloudstack-common/vms/systemvm.iso > > > > > > baba18f156395da3a5d8208727d8f421 > > > > > > /usr/share/cloudstack-common/vms/systemvm.iso > > > > > > > > > > > > Also the private key on xenserver root and management server are > > > same. > > > > > > > > > > > > management server > > > > > > /usr/share/cloudstack-common/scripts/vm/systemvm/id_rsa.cloud > > > > > > > > > > > > xenserver path > > > > > > /root/.ssh/id_rsa.cloud > > > > > > > > > > > > - Ammad Ali > > > > > > > > > > > > On Thu, Aug 13, 2020 at 11:27 AM Ammad Syed < > syedamma...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > >> Here is the link for download management logs. > > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > > > > > > https://drive.google.com/file/d/1l6HDPguGUNaOxsc7VSaj7eaP0F2UOFjA/view?usp=sharing > > > > > >> > > > > > >> On Thu, Aug 13, 2020 at 11:22 AM Ammad Syed < > > syedamma...@gmail.com> > > > > > >> wrote: > > > > > >> > > > > > >>> I have reverted the version back to 4.11.3. But I have saved > logs > > > > > >>> starting from upgrade. > > > > > >>> > > > > > >>> I think the key has been copied successfully in system vm iso. > > > > > >>> > > > > > >>> 2020-07-25 00:34:17,214 INFO [c.c.s.ConfigurationServerImpl] > > > > > >>> (main:null) (logid:) Going to update systemvm iso with > generated > > > > > keypairs > > > > > >>> if needed > > > > > >>> 2020-07-25 00:34:17,214 INFO [c.c.s.ConfigurationServerImpl] > > > > > >>> (main:null) (logid:) Trying to inject public and private keys > > into > > > > > systemvm > > > > > >>> iso > > > > > >>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null) > > (logid:) > > > > > >>> Looking for vms/systemvm.iso in the classpath > > > > > >>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null) > > (logid:) > > > > > >>> System resource: > > file:/usr/share/cloudstack-common/vms/systemvm.iso > > > > > >>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null) > > (logid:) > > > > > >>> Absolute path = /usr/share/cloudstack-common/vms/systemvm.iso > > > > > >>> 2020-07-25 00:34:17,218 DEBUG [c.c.s.ConfigurationServerImpl] > > > > > >>> (main:null) (logid:) Executing: /bin/bash > > > > > >>> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh > > > > > >>> /var/cloudstack/management/.ssh/id_rsa.pub > > > > > >>> /var/cloudstack/management/.ssh/id_rsa > > > > > >>> /usr/share/cloudstack-common/vms/systemvm.iso > > > > > >>> 2020-07-25 00:34:17,636 INFO [c.c.s.ConfigurationServerImpl] > > > > > >>> (main:null) (logid:) Injected public and private keys into > > systemvm > > > > iso > > > > > >>> with result : null > > > > > >>> 2020-07-25 00:34:50,613 DEBUG [c.c.h.x.r.CitrixResourceBase] > > > > > >>> (DirectAgent-1:ctx-d3dc4cf2) (logid:1d22396d) Copying > > > > > >>> > > > > > > > > > > > > > > > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso > > > > > >>> to /opt/xensource/packages/iso on 172.16.2.22 with permission > > 0644 > > > > > >>> 2020-07-25 00:34:52,566 DEBUG [c.c.h.x.r.CitrixResourceBase] > > > > > >>> (DirectAgent-2:ctx-2537b610) (logid:29c67b7a) Copying > > > > > >>> > > > > > > > > > > > > > > > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso > > > > > >>> to /opt/xensource/packages/iso on 172.16.2.5 with permission > 0644 > > > > > >>> 2020-07-25 00:34:53,170 DEBUG [c.c.h.x.r.CitrixResourceBase] > > > > > >>> (DirectAgent-3:ctx-168ac27d) (logid:a7862c4b) Copying > > > > > >>> > > > > > > > > > > > > > > > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso > > > > > >>> to /opt/xensource/packages/iso on 172.16.2.9 with permission > 0644 > > > > > >>> 2020-07-25 00:34:54,621 DEBUG [c.c.h.x.r.CitrixResourceBase] > > > > > >>> (DirectAgent-6:ctx-f640fe55) (logid:0a62d4cf) Copying > > > > > >>> > > > > > > > > > > > > > > > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso > > > > > >>> to /opt/xensource/packages/iso on 172.16.2.5 with permission > 0644 > > > > > >>> > > > > > >>> I have attached complete management logs. The start of logs is > > the > > > > > start > > > > > >>> of management server after upgrade of management server from > > 4.11.3 > > > > to > > > > > >>> 4.13.1. > > > > > >>> > > > > > >>> Ammad Ali > > > > > >>> > > > > > >>> On Thu, Aug 13, 2020 at 1:35 AM Andrija Panic < > > > > andrija.pa...@gmail.com > > > > > > > > > > > >>> wrote: > > > > > >>> > > > > > >>>> Do you get an error while trying to inject ssh key into the > > > > > systemvm.iso > > > > > >>>> (mgmt logs) , or can you confirm that the systemvm.iso on XS > > host > > > > and > > > > > >>>> the > > > > > >>>> one on the mgmt server are identical, md5sum them (i.e. has > the > > > iso > > > > > been > > > > > >>>> copied over to the XS successfully) - this might explain not > > being > > > > > able > > > > > >>>> to > > > > > >>>> login via ssh with your private key. > > > > > >>>> > > > > > >>>> Best, > > > > > >>>> > > > > > >>>> On Wed, 12 Aug 2020, 09:08 Ammad Syed, <syedamma...@gmail.com > > > > > > wrote: > > > > > >>>> > > > > > >>>> > Yes this is exactly the same issue that i faced. > > > > > >>>> > > > > > > >>>> > Sent from my iPhone > > > > > >>>> > > > > > > >>>> > > On 12-Aug-2020, at 8:35 AM, Eric Lee Green < > > > > > >>>> eric.lee.gr...@gmail.com> > > > > > >>>> > wrote: > > > > > >>>> > > > > > > > >>>> > > Correct, 4.11.3 template is used for 4.11.3, 4.12, and > > 4.13. > > > > 4.14 > > > > > >>>> moves > > > > > >>>> > to the 4.14.0 template. > > > > > >>>> > > > > > > > >>>> > > There seems to be something odd happening key-wise > sometimes > > > > with > > > > > >>>> > upgrades from 4.11.3 to 4.13.1 or 4.14.0. I managed an > > upgrade > > > > > from > > > > > >>>> > 4.11.3 to 4.13.1 that *almost* worked, but the secondary > > storage > > > > VM > > > > > >>>> > wouldn't work and thus I couldn't spawn new virtual > machines. > > > Same > > > > > >>>> symptom > > > > > >>>> > -- key error when the agent tried to ssh into it. And > deleting > > > it > > > > > and > > > > > >>>> > making it respawn didn't help. Then I tried 4.11.3 to 4.14.0 > > and > > > > > >>>> *all* the > > > > > >>>> > VM's failed at that point (of course, that was with the new > > > > > template). > > > > > >>>> > > > > > > > >>>> > > Right now I'm back at 4.11.3 until this can be figured > out. > > > > > >>>> > > > > > > > >>>> > >> On 8/11/2020 5:53 AM, Ammad Syed wrote: > > > > > >>>> > >> Hi, > > > > > >>>> > >> > > > > > >>>> > >> I think 4.12 and 4.13 uses same systemVM template i.e > > 4.11.3 > > > > > >>>> version, > > > > > >>>> > which > > > > > >>>> > >> I already have registered. Currently I am running 4.11.3 > > > > version > > > > > >>>> of ACS. > > > > > >>>> > >> > > > > > >>>> > >> MariaDB [cloud]> SELECT id,name,type,cross_zones,state > FROM > > > > > >>>> > >> cloud.vm_template WHERE name like '%systemvm-xenserver%' > > AND > > > > > >>>> removed IS > > > > > >>>> > >> NULL; > > > > > >>>> > >> > > > > > >>>> > > > > > > > > > +------+-----------------------------+--------+-------------+----------+ > > > > > >>>> > >> | id | name | type | > > cross_zones | > > > > > >>>> state | > > > > > >>>> > >> > > > > > >>>> > > > > > > > > > +------+-----------------------------+--------+-------------+----------+ > > > > > >>>> > >> | 337 | systemvm-xenserver-3.0.0 | SYSTEM | > > 0 | > > > > > >>>> Inactive | > > > > > >>>> > >> | 418 | systemvm-xenserver-4.2 | SYSTEM | > > 0 | > > > > > >>>> Active | > > > > > >>>> > >> | 472 | systemvm-xenserver-4.3 | USER | > > 1 | > > > > > >>>> Inactive | > > > > > >>>> > >> | 473 | systemvm-xenserver-4.3 | USER | > > 1 | > > > > > >>>> Inactive | > > > > > >>>> > >> | 474 | systemvm-xenserver-4.3 | USER | > > 1 | > > > > > >>>> Inactive | > > > > > >>>> > >> | 475 | systemvm-xenserver-4.3 | USER | > > 1 | > > > > > >>>> Inactive | > > > > > >>>> > >> | 476 | systemvm-xenserver-4.3 | USER | > > 0 | > > > > > >>>> Inactive | > > > > > >>>> > >> | 479 | systemvm-xenserver-4.3-2 | USER | > > 1 | > > > > > >>>> Inactive | > > > > > >>>> > >> | 480 | systemvm-xenserver-4.3 | SYSTEM | > > 0 | > > > > > >>>> Active | > > > > > >>>> > >> | 549 | systemvm-xenserver-4.5.1 | USER | > > 0 | > > > > > >>>> Active | > > > > > >>>> > >> | 550 | systemvm-xenserver-4.5.1 | SYSTEM | > > 0 | > > > > > >>>> Active | > > > > > >>>> > >> | 651 | systemvm-xenserver-4.7.0 | USER | > > 0 | > > > > > >>>> Inactive | > > > > > >>>> > >> | 652 | systemvm-xenserver-4.7.0 | USER | > > 0 | > > > > > >>>> Inactive | > > > > > >>>> > >> | 653 | systemvm-xenserver-4.7.0 | SYSTEM | > > 0 | > > > > > >>>> Inactive | > > > > > >>>> > >> | 737 | systemvm-xenserver-4.9.2 | SYSTEM | > > 1 | > > > > > >>>> Inactive | > > > > > >>>> > >> | 739 | systemvm-xenserver-4.9.2-sb | SYSTEM | > > 1 | > > > > > >>>> Active | > > > > > >>>> > >> | 1245 | systemvm-xenserver-4.11.1 | SYSTEM | > > 1 | > > > > > >>>> Active | > > > > > >>>> > >> | 1584 | systemvm-xenserver-4.11.2 | SYSTEM | > > 1 | > > > > > >>>> Active | > > > > > >>>> > >> | 1677 | systemvm-xenserver-4.11.3 | SYSTEM | > > 1 | > > > > > >>>> Active | > > > > > >>>> > >> > > > > > >>>> > > > > > > > > > +------+-----------------------------+--------+-------------+----------+ > > > > > >>>> > >> > > > > > >>>> > >> - Ammad > > > > > >>>> > >> > > > > > >>>> > >> On Tue, Aug 11, 2020 at 5:17 PM Pierre-Luc Dion < > > > > > >>>> pdion...@apache.org> > > > > > >>>> > wrote: > > > > > >>>> > >> db. > > > > > >>>> > >>> Hi Syed, > > > > > >>>> > >>> From 4.12, the systemvm template had to be upgraded > > because > > > of > > > > > OS > > > > > >>>> > change in > > > > > >>>> > >>> the template, moved to a latest version of Debian. > Because > > > of > > > > > >>>> that, > > > > > >>>> > some VR > > > > > >>>> > >>> scripts have changed and make obsolete older version of > > VRs, > > > > so > > > > > >>>> you > > > > > >>>> > will > > > > > >>>> > >>> most likely have to register an updated systemvm > templates > > > and > > > > > >>>> upgrade > > > > > >>>> > your > > > > > >>>> > >>> system VMs and VRs. > > > > > >>>> > >>> > > > > > >>>> > >>> Regards, > > > > > >>>> > >>> > > > > > >>>> > >>>> On Tue, Aug 11, 2020 at 6:24 AM Ammad Syed < > > > > > >>>> syedamma...@gmail.com> > > > > > >>>> > wrote: > > > > > >>>> > >>> > > > > > >>>> > >>>> Hi Guys, > > > > > >>>> > >>>> > > > > > >>>> > >>>> I was previously on 4.9.3 cloudstack and upgraded to > > 4.11.1 > > > > > then > > > > > >>>> > 4.11.3. > > > > > >>>> > >>>> The version 4.11.3 is working fine since six months. > > > > > >>>> > >>>> > > > > > >>>> > >>>> Now I have tried to upgrade my system from 4.11.3 to > > > 4.13.1. > > > > > The > > > > > >>>> > upgrade > > > > > >>>> > >>>> goes successful. I didn't uploaded any system VM > > template. > > > > > >>>> However the > > > > > >>>> > >>>> problem occured when I recreated my systemVM of POD, > the > > VM > > > > > >>>> recreated > > > > > >>>> > and > > > > > >>>> > >>>> its state was running but agent state was not getting > up, > > > its > > > > > >>>> showing > > > > > >>>> > >>> blank > > > > > >>>> > >>>> in column. > > > > > >>>> > >>>> > > > > > >>>> > >>>> Digging further via job logs, the job is failed with > > error > > > > that > > > > > >>>> > unable to > > > > > >>>> > >>>> execute command via ssh. Below are the logs. > > > > > >>>> > >>>> > > > > > >>>> > >>>> 2020-07-25 02:30:48,126 ERROR [c.c.u.s.SshHelper] > > > > > >>>> > >>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) SSH > > > execution > > > > > of > > > > > >>>> > command > > > > > >>>> > >>>> /opt/cloud/bin/router_proxy.sh keystore-s > > > > > >>>> > >>>> etup 169.254.2.199 > > > > > >>>> /usr/local/cloud/systemvm/conf/agent.properties > > > > > >>>> > >>>> /usr/local/cloud/systemvm/conf/cloud.jks > TJaQYChYBwKh7Cx9 > > > 365 > > > > > >>>> > >>>> /usr/local/cloud/systemvm/conf/clou > > > > > >>>> > >>>> d.csr has an error status code in return. Result > output: > > > > > >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG > > [c.c.a.m.DirectAgentAttache] > > > > > >>>> > >>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq > > > > > >>>> > >>>> 906-3195585410596077730: Response Received: > > > > > >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request] > > > > > >>>> > >>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq > > > > > >>>> > >>>> 906-3195585410596077730: Processing: { Ans: , MgmtId: > > > > > 779271079 > > > > > >>>> > >>>> 43497, via: 906(xen-21-10-a3-khi02), Ver: v1, Flags: > 10, > > > > > >>>> > >>>> [{"org.apache.cloudstack.ca > > > > > >>>> > >>>> .SetupKeystoreAnswer":{"result":false,"wait":0}}] > > > > > >>>> > >>>> } > > > > > >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request] > > > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d > > job-1208155/job-1208258 > > > > > >>>> > ctx-df740f75) > > > > > >>>> > >>>> (logid:9fa7dece) Seq 906-319558541059607773 > > > > > >>>> > >>>> 0: Received: { Ans: , MgmtId: 77927107943497, via: > > > > > >>>> > >>>> 906(xen-21-10-a3-khi02), Ver: v1, Flags: 10, { > > > > > >>>> SetupKeystoreAnswer } } > > > > > >>>> > >>>> 2020-07-25 02:30:48,127 ERROR > > > > [c.c.v.VirtualMachineManagerImpl] > > > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d > > job-1208155/job-1208258 > > > > > >>>> > ctx-df740f75) > > > > > >>>> > >>>> (logid:9fa7dece) Failed to setup keystore and generate > > CSR > > > > for > > > > > >>>> system > > > > > >>>> > vm: > > > > > >>>> > >>>> s-24142-VM > > > > > >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG > > [c.c.v.VmWorkJobHandlerProxy] > > > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d > > job-1208155/job-1208258 > > > > > >>>> > ctx-df740f75) > > > > > >>>> > >>>> (logid:9fa7dece) Done executing VM work job: > > > > > >>>> > >>>> > > > > > >>>> > >>>> > > > > > >>>> > >>> > > > > > >>>> > > > > > > >>>> > > > > > > > > > > > > > > > com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":24142,"handlerName":"VirtualMachineManagerImpl"} > > > > > >>>> > >>>> 2020-07-25 02:30:48,128 DEBUG > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d > > job-1208155/job-1208258 > > > > > >>>> > ctx-df740f75) > > > > > >>>> > >>>> (logid:9fa7dece) Complete async job-1208258, jobStatus: > > > > > >>>> SUCCEEDED, > > > > > >>>> > >>>> resultCode: 0, result: null > > > > > >>>> > >>>> > > > > > >>>> > >>>> I tried to dig it further, I was unable to login > systemVM > > > via > > > > > >>>> ssh from > > > > > >>>> > >>>> xenserver host with key /root/.ssh/id_rsa.cloud placed. > > > Look > > > > > like > > > > > >>>> > private > > > > > >>>> > >>>> key issue. However I am able to login on my old > > systemVMs ( > > > > i.e > > > > > >>>> > created > > > > > >>>> > >>> on > > > > > >>>> > >>>> ACS 4.11.3) > > > > > >>>> > >>>> > > > > > >>>> > >>>> Also I have SSL certificate enabled for console proxy > on > > my > > > > ACS > > > > > >>>> 4.11.3 > > > > > >>>> > >>> and > > > > > >>>> > >>>> I am using only xenserver 7.0 hosts. > > > > > >>>> > >>>> > > > > > >>>> > >>>> I tried to disable SSL on secstorage and console proxy > > from > > > > > >>>> global > > > > > >>>> > >>>> settings, but still didn't worked. > > > > > >>>> > >>>> > > > > > >>>> > >>>> I had a fresh installation of ACS 4.13.1 with xenserver > > > 7.0, > > > > > >>>> systemVMs > > > > > >>>> > >>> are > > > > > >>>> > >>>> working fine in it. > > > > > >>>> > >>>> > > > > > >>>> > >>>> Please advise. > > > > > >>>> > >>>> -- > > > > > >>>> > >>>> Regards, > > > > > >>>> > >>>> > > > > > >>>> > >>>> > > > > > >>>> > >>>> Syed Ammad Ali > > > > > >>>> > >>>> > > > > > >>>> > >> > > > > > >>>> > > > > > > >>>> > > > > > >>> > > > > > >>> > > > > > >>> -- > > > > > >>> Regards, > > > > > >>> > > > > > >>> > > > > > >>> Syed Ammad Ali > > > > > >>> > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> Regards, > > > > > >> > > > > > >> > > > > > >> Syed Ammad Ali > > > > > >> > > > > > > > > > > > > > > > > > > -- > > > > > > Regards, > > > > > > > > > > > > > > > > > > Syed Ammad Ali > > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > > > > > > > > > > > Syed Ammad Ali > > > > > > > > > > > > > > > > > -- > > > > > > > > Andrija Panić > > > > > > > > > > > > > -- > > > Regards, > > > > > > > > > Syed Ammad Ali > > > > > > > > > -- > > > > Andrija Panić > > > > > -- > Regards, > > > Syed Ammad Ali > -- Andrija Panić
