Hi Guys, I was previously on 4.9.3 cloudstack and upgraded to 4.11.1 then 4.11.3. The version 4.11.3 is working fine since six months.
Now I have tried to upgrade my system from 4.11.3 to 4.13.1. The upgrade goes successful. I didn't uploaded any system VM template. However the problem occured when I recreated my systemVM of POD, the VM recreated and its state was running but agent state was not getting up, its showing blank in column. Digging further via job logs, the job is failed with error that unable to execute command via ssh. Below are the logs. 2020-07-25 02:30:48,126 ERROR [c.c.u.s.SshHelper] (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) SSH execution of command /opt/cloud/bin/router_proxy.sh keystore-s etup 169.254.2.199 /usr/local/cloud/systemvm/conf/agent.properties /usr/local/cloud/systemvm/conf/cloud.jks TJaQYChYBwKh7Cx9 365 /usr/local/cloud/systemvm/conf/clou d.csr has an error status code in return. Result output: 2020-07-25 02:30:48,127 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq 906-3195585410596077730: Response Received: 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request] (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq 906-3195585410596077730: Processing: { Ans: , MgmtId: 779271079 43497, via: 906(xen-21-10-a3-khi02), Ver: v1, Flags: 10, [{"org.apache.cloudstack.ca.SetupKeystoreAnswer":{"result":false,"wait":0}}] } 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request] (Work-Job-Executor-41:ctx-4e3c666d job-1208155/job-1208258 ctx-df740f75) (logid:9fa7dece) Seq 906-319558541059607773 0: Received: { Ans: , MgmtId: 77927107943497, via: 906(xen-21-10-a3-khi02), Ver: v1, Flags: 10, { SetupKeystoreAnswer } } 2020-07-25 02:30:48,127 ERROR [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-41:ctx-4e3c666d job-1208155/job-1208258 ctx-df740f75) (logid:9fa7dece) Failed to setup keystore and generate CSR for system vm: s-24142-VM 2020-07-25 02:30:48,127 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-41:ctx-4e3c666d job-1208155/job-1208258 ctx-df740f75) (logid:9fa7dece) Done executing VM work job: com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":24142,"handlerName":"VirtualMachineManagerImpl"} 2020-07-25 02:30:48,128 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-41:ctx-4e3c666d job-1208155/job-1208258 ctx-df740f75) (logid:9fa7dece) Complete async job-1208258, jobStatus: SUCCEEDED, resultCode: 0, result: null I tried to dig it further, I was unable to login systemVM via ssh from xenserver host with key /root/.ssh/id_rsa.cloud placed. Look like private key issue. However I am able to login on my old systemVMs ( i.e created on ACS 4.11.3) Also I have SSL certificate enabled for console proxy on my ACS 4.11.3 and I am using only xenserver 7.0 hosts. I tried to disable SSL on secstorage and console proxy from global settings, but still didn't worked. I had a fresh installation of ACS 4.13.1 with xenserver 7.0, systemVMs are working fine in it. Please advise. -- Regards, Syed Ammad Ali