Hi Ammad, Could you try re-injecting the keys into the systemVM, as this MAY happen due to stale id_rsa keys. Run the injectkeys.sh to inject the existing public ssh key into the authorized_keys file inside the systemvm.iso. Execute:
/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh /root/.ssh/id_rsa.pub /root/.ssh/id_rsa /usr/share/cloudstack-common/vms/systemvm.iso' * Then replace the systemvm.iso on the hypervisor hosts. For each host: 1. Login and type command - xe host-param-clear param-name=tags uuid=<uuid of the XS host> host uuid maybe obtained using: xe host-list 2. 2. Restart MS and the iso should get propagated for all. Check timestamp. 3. 3. Restart systemvms and they should get the iso inserted. 4. https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso#SystemVm.iso-Xen Thanks, Pearl ________________________________ From: Ammad Syed <syedamma...@gmail.com> Sent: Wednesday, September 2, 2020 9:12 PM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error Guys, please advise how to troubleshoot this further ? How can i enable trace level debugging ? any help in this will be highly appreciated? Ammad pearl.dsi...@shapeblue.com www.shapeblue.com 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue > On 27-Aug-2020, at 1:41 PM, Vivek Kumar <vivek.ku...@indiqus.com.invalid> > wrote: > > Pardon, the whole mail chain was not visible so replied accordingly. It > seems like you are doing SSH using root shell so it doesn’t matter. > > Vivek Kumar > Manager - Cloud & DevOps > IndiQus Technologies > 24*7 O +91 11 4055 1411 | M +91 7503460090 > www.indiqus.com<http://www.indiqus.com> <http://indiqus.com/> > > This message is intended only for the use of the individual or entity to > which it is addressed and may contain information that is confidential and/or > privileged. If you are not the intended recipient please delete the original > message and any copy of it from your computer system. You are hereby notified > that any dissemination, distribution or copying of this communication is > strictly prohibited unless proper authorization has been obtained for such > action. If you have received this communication in error, please notify the > sender immediately. Although IndiQus attempts to sweep e-mail and attachments > for viruses, it does not guarantee that both are virus-free and accepts no > liability for any damage sustained as a result of viruses. > >> On 27-Aug-2020, at 2:03 PM, Vivek Kumar <vivek.ku...@indiqus.com> wrote: >> >> Hello Ammad, >> >> You haven’t defined the user while logging to the system VM through the Link >> local IP that’s why you have got the error while logging to the system VM. >> >>>> ssh -i /root/.ssh/id_rsa.cloud 169.254.0.121 -p 3922 >> >> Try instead below >> >>>> ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.0.121 >>>> <mailto:root@169.254.0.121> >> >> Vivek Kumar >> Manager - Cloud & DevOps >> IndiQus Technologies >> 24*7 O +91 11 4055 1411 | M +91 7503460090 >> www.indiqus.com<http://www.indiqus.com> <http://indiqus.com/> >> >> This message is intended only for the use of the individual or entity to >> which it is addressed and may contain information that is confidential >> and/or privileged. If you are not the intended recipient please delete the >> original message and any copy of it from your computer system. You are >> hereby notified that any dissemination, distribution or copying of this >> communication is strictly prohibited unless proper authorization has been >> obtained for such action. If you have received this communication in error, >> please notify the sender immediately. Although IndiQus attempts to sweep >> e-mail and attachments for viruses, it does not guarantee that both are >> virus-free and accepts no liability for any damage sustained as a result of >> viruses. >> >>>> On 27-Aug-2020, at 1:59 PM, Ammad Syed <syedamma...@gmail.com >>>> <mailto:syedamma...@gmail.com>> wrote: >>> >>> Guys, any help to troubleshoot this further would be highly appreciated. >>> >>> >>> Ammad Ali >>>> On 18-Aug-2020, at 1:05 PM, Ammad Syed <syedamma...@gmail.com >>>> <mailto:syedamma...@gmail.com>> wrote: >>>> >>>> >>>> Hi, >>>> >>>> The systemVM is accessible with 169.x.x.x IP from xenserver host. But >>>> login with private key is denied. I am testing this on my test environment. >>>> >>>> [root@xenserver ~]# ping 169.254.0.121 >>>> PING 169.254.0.121 (169.254.0.121) 56(84) bytes of data. >>>> 64 bytes from 169.254.0.121: icmp_seq=1 ttl=64 time=0.328 ms >>>> 64 bytes from 169.254.0.121: icmp_seq=2 ttl=64 time=0.124 ms >>>> 64 bytes from 169.254.0.121: icmp_seq=3 ttl=64 time=0.111 ms >>>> 64 bytes from 169.254.0.121: icmp_seq=4 ttl=64 time=0.126 ms >>>> ^C >>>> --- 169.254.0.121 ping statistics --- >>>> 4 packets transmitted, 4 received, 0% packet loss, time 2997ms >>>> rtt min/avg/max/mdev = 0.111/0.172/0.328/0.090 ms >>>> [root@xenserver ~]# >>>> [root@xenserver ~]# >>>> [root@xenserver ~]# ssh -i /root/.ssh/id_rsa.cloud 169.254.0.121 -p 3922 >>>> Permission denied (publickey). >>>> >>>> I have checked by logging into the systemVM, the public key is not found >>>> in /root/.ssh/authorized_keys. >>>> >>>> I have digged further, the systemvm.iso is same on the management node and >>>> xenserver host. I have checked by mounting the ISO as well, the public key >>>> is there in authorized_keys in ISO. >>>> >>>> Restarting / recreating systemVM isn't fixed the issue. >>>> >>>> The systemVM is already running on HVM mode on xenserver host as the >>>> template OS type is Debian 8 64bit is selected. >>>> >>>> On my PRD upgrade, I have tested on one zone only. Currently I am testing >>>> on my test ACS environment which have only one zone. >>>> >>>> Ammad Ali >>>> >>>>> On Mon, Aug 17, 2020 at 11:10 PM Andrija Panic <andrija.pa...@gmail.com >>>>> <mailto:andrija.pa...@gmail.com>> wrote: >>>>> It was my tool (GUI based tool, that still doesn't return any line (???) >>>>> but I see it. >>>>> >>>>> Problem is with 169.254.1.178 not being reachable over SSH (2020-07-25 >>>>> 01:45:52,097 DEBUG [c.c.h.x.r.CitrixResourceBase] >>>>> (DirectAgent-57:ctx-ded7f7f9) (logid:a75971c2) Executing command in VR: >>>>> /opt/cloud/bin/router_proxy.sh keystore-setup 169.254.1.178 >>>>> /usr/local/cloud/systemvm/conf/agent.properties >>>>> /usr/local/cloud/systemvm/conf/cloud.jks Xe8zzqZmUp9wmABH 365 >>>>> /usr/local/cloud/systemvm/conf/cloud.csr) >>>>> >>>>> Can you try to ssh from that particular XenServer (where the SSVM is >>>>> running) to the IP 169.xxxxx on port 3922 (ssh port) using the private key >>>>> - if you log in to the VM via it's console (root/password) - do you see >>>>> with ifconfig the interface being started with 169..xxxx IP address? I >>>>> assume that destroying the SSVM and starting a new one doesn't fix the >>>>> issue? >>>>> >>>>> You can also try changing the OS type of the systemVM template to Other >>>>> Linux 64bit, so it runs in HVM mode (instead of PV) - you'll need to see >>>>> ID >>>>> for that OS type from the guest_os table, and set the guest_os_id field in >>>>> the vm_template table for your systemVM template. >>>>> >>>>> Does the issue occur in other zones as well? (assuming same hypervisor >>>>> type) >>>>> >>>>> >>>>> On Mon, 17 Aug 2020 at 15:31, Ammad Syed <syedamma...@gmail.com >>>>> <mailto:syedamma...@gmail.com>> wrote: >>>>> >>>>>> I have rechecked the logs are there. Please review below lines. >>>>>> >>>>>> >>>>>> >>>>>> C:\Users\ammad\Downloads\management-server.log-4.13\management-server.log-4.13.1-errors >>>>>> (27 hits) >>>>>> Line 409969: 2020-07-25 01:45:52,320 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-58:ctx-9dc75334 >>>>>> job-1207969/job-1208104 ctx-09e752ec) (logid:a75971c2) Failed to setup >>>>>> keystore and generate CSR for system vm: s-25023-VM >>>>>> Line 437413: 2020-07-25 01:51:33,845 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-70:ctx-457ce4ee >>>>>> job-1207969/job-1208129 ctx-195da69e) (logid:a75971c2) Failed to setup >>>>>> keystore and generate CSR for system vm: s-25024-VM >>>>>> Line 549538: 2020-07-25 02:06:44,833 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-3:ctx-4634f7e0 >>>>>> job-1208154/job-1208161 ctx-05d065dc) (logid:91fc037a) Failed to setup >>>>>> keystore and generate CSR for system vm: v-25027-VM >>>>>> Line 663111: 2020-07-25 02:23:43,783 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-22:ctx-fe0b857b >>>>>> job-1208154/job-1208222 ctx-ef088cd7) (logid:91fc037a) Failed to setup >>>>>> keystore and generate CSR for system vm: v-20771-VM >>>>>> Line 667060: 2020-07-25 02:24:30,221 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-23:ctx-662cfe5d >>>>>> job-1208154/job-1208229 ctx-e59878e9) (logid:91fc037a) Failed to setup >>>>>> keystore and generate CSR for system vm: v-24763-VM >>>>>> Line 674269: 2020-07-25 02:25:47,345 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-24:ctx-b6529613 >>>>>> job-1208154/job-1208233 ctx-7da81cab) (logid:91fc037a) Failed to setup >>>>>> keystore and generate CSR for system vm: v-20773-VM >>>>>> Line 678169: 2020-07-25 02:26:20,355 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-31:ctx-7a062a66 >>>>>> job-1208154/job-1208244 ctx-4f48d08a) (logid:91fc037a) Failed to setup >>>>>> keystore and generate CSR for system vm: v-25027-VM >>>>>> Line 697113: 2020-07-25 02:30:09,669 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-40:ctx-3be9fddc >>>>>> job-1208154/job-1208257 ctx-ed08fc43) (logid:91fc037a) Failed to setup >>>>>> keystore and generate CSR for system vm: v-20771-VM >>>>>> Line 701161: 2020-07-25 02:30:48,127 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-41:ctx-4e3c666d >>>>>> job-1208155/job-1208258 ctx-df740f75) (logid:9fa7dece) Failed to setup >>>>>> keystore and generate CSR for system vm: s-24142-VM >>>>>> Line 701838: 2020-07-25 02:30:56,759 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-42:ctx-04ac8e71 >>>>>> job-1208154/job-1208259 ctx-36c67b5e) (logid:91fc037a) Failed to setup >>>>>> keystore and generate CSR for system vm: v-24763-VM >>>>>> Line 703865: 2020-07-25 02:31:25,123 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-46:ctx-24bdbe8f >>>>>> job-1208154/job-1208266 ctx-fa500d3c) (logid:91fc037a) Failed to setup >>>>>> keystore and generate CSR for system vm: v-20773-VM >>>>>> Line 712078: 2020-07-25 02:32:28,666 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-4d5db5c2 >>>>>> job-1208155/job-1208269 ctx-b0f67c2e) (logid:9fa7dece) Failed to setup >>>>>> keystore and generate CSR for system vm: s-25033-VM >>>>>> Line 715204: 2020-07-25 02:33:02,875 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-51:ctx-594da6d2 >>>>>> job-1208155/job-1208273 ctx-ca115800) (logid:9fa7dece) Failed to setup >>>>>> keystore and generate CSR for system vm: s-25035-VM >>>>>> Line 743140: 2020-07-25 02:38:52,257 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-59:ctx-a3f032da >>>>>> job-1208155/job-1208294 ctx-626a2716) (logid:9fa7dece) Failed to setup >>>>>> keystore and generate CSR for system vm: s-25037-VM >>>>>> Line 817646: 2020-07-25 02:47:07,127 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-5:ctx-ca073bd1 >>>>>> job-1208297/job-1208305 ctx-6213af3b) (logid:1a2a782c) Failed to setup >>>>>> keystore and generate CSR for system vm: s-25038-VM >>>>>> Line 847150: 2020-07-25 02:51:46,293 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-8:ctx-9db8c548 >>>>>> job-1208296/job-1208312 ctx-83c570ff) (logid:436d1800) Failed to setup >>>>>> keystore and generate CSR for system vm: v-25040-VM >>>>>> Line 903949: 2020-07-25 02:57:48,515 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-4:ctx-f35fd59a >>>>>> job-1208320/job-1208333 ctx-f81179ac) (logid:05eacf0d) Failed to setup >>>>>> keystore and generate CSR for system vm: v-25041-VM >>>>>> Line 957628: 2020-07-25 03:04:13,286 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-24:ctx-dc18956a >>>>>> job-1208320/job-1208381 ctx-b53f5d7c) (logid:05eacf0d) Failed to setup >>>>>> keystore and generate CSR for system vm: v-20771-VM >>>>>> Line 961606: 2020-07-25 03:05:00,522 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-28:ctx-83e48421 >>>>>> job-1208320/job-1208392 ctx-e6767f04) (logid:05eacf0d) Failed to setup >>>>>> keystore and generate CSR for system vm: v-24763-VM >>>>>> Line 966633: 2020-07-25 03:05:29,767 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-30:ctx-897e2a66 >>>>>> job-1208320/job-1208397 ctx-39c396e1) (logid:05eacf0d) Failed to setup >>>>>> keystore and generate CSR for system vm: v-20773-VM >>>>>> Line 968892: 2020-07-25 03:05:53,648 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-36:ctx-68e45b80 >>>>>> job-1208320/job-1208405 ctx-1aaf96dd) (logid:05eacf0d) Failed to setup >>>>>> keystore and generate CSR for system vm: v-25041-VM >>>>>> Line 1003036: 2020-07-25 03:13:33,582 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-45:ctx-0f552e0e >>>>>> job-1208320/job-1208422 ctx-e8835fe9) (logid:05eacf0d) Failed to setup >>>>>> keystore and generate CSR for system vm: v-20771-VM >>>>>> Line 1006496: 2020-07-25 03:14:09,787 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-46:ctx-08e5db24 >>>>>> job-1208320/job-1208423 ctx-2cb636c2) (logid:05eacf0d) Failed to setup >>>>>> keystore and generate CSR for system vm: v-24763-VM >>>>>> Line 1008408: 2020-07-25 03:14:39,584 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-47:ctx-8b9288a9 >>>>>> job-1208320/job-1208424 ctx-c05d1544) (logid:05eacf0d) Failed to setup >>>>>> keystore and generate CSR for system vm: v-20773-VM >>>>>> Line 1023914: 2020-07-25 03:15:58,139 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-4d54d717 >>>>>> job-1208320/job-1208425 ctx-27e5397d) (logid:05eacf0d) Failed to setup >>>>>> keystore and generate CSR for system vm: v-25046-VM >>>>>> Line 1089568: 2020-07-25 03:24:24,534 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-59:ctx-62ff8844 >>>>>> job-1208320/job-1208449 ctx-d6a14f7f) (logid:05eacf0d) Failed to setup >>>>>> keystore and generate CSR for system vm: v-25048-VM >>>>>> Line 1097605: 2020-07-25 03:25:23,179 ERROR >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-62:ctx-24407a9e >>>>>> job-1208321/job-1208452 ctx-84f80ffa) (logid:42edff3a) Failed to setup >>>>>> keystore and generate CSR for system vm: s-25050-VM >>>>>> >>>>>> -Ammad Ali >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Aug 17, 2020 at 5:56 PM Andrija Panic <andrija.pa...@gmail.com >>>>>> <mailto:andrija.pa...@gmail.com>> >>>>>> wrote: >>>>>> >>>>>>> Ammad, no such log lines in the file you uploaded....? >>>>>>> >>>>>>> On Mon, 17 Aug 2020 at 09:02, Ammad Syed <syedamma...@gmail.com >>>>>>> <mailto:syedamma...@gmail.com>> wrote: >>>>>>> >>>>>>>> Hi Andrija, >>>>>>>> >>>>>>>> The error occurred only one time. Can you please try to grep "Failed to >>>>>>>> setup keystore and generate CSR" you will see all the systemVMs that I >>>>>>>> tried to create failed because of certificate generation failed. OR you >>>>>>> can >>>>>>>> grep job-1208321/job-1208452 this job as well. >>>>>>>> >>>>>>>> Also I mentioned above that same thing is happening in my test >>>>>>> environment >>>>>>>> that I created 4.11.1 then upgraded 4.11.3 > 4.13.1. >>>>>>>> >>>>>>>> Ammad Ali >>>>>>>> >>>>>>>> On Mon, Aug 17, 2020 at 11:17 AM Andrija Panic < >>>>>> andrija.pa...@gmail.com <mailto:andrija.pa...@gmail.com>> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> 2020-07-25 03:23:51,093 WARN [c.c.v.SystemVmLoadScanner] >>>>>>>>> (secstorage-1:ctx-50756e9b) (logid:3d7c25cb) Unexpected exception >>>>>>> Failed >>>>>>>> to >>>>>>>>> fetch any free public IP address >>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to fetch any >>>>>>> free >>>>>>>>> public IP address >>>>>>>>> >>>>>>>>> ? >>>>>>>>> >>>>>>>>> On Sat, 15 Aug 2020 at 18:05, Ammad Syed <syedamma...@gmail.com >>>>>>>>> <mailto:syedamma...@gmail.com>> >>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I have checked further, the key is successfully injected in >>>>>>>> systemvm.iso >>>>>>>>> on >>>>>>>>>> the management node and copied successfully to xenserver. I have >>>>>>>> checked >>>>>>>>> on >>>>>>>>>> systemVM there is no public key found in authorized_keys of root. >>>>>>>>>> >>>>>>>>>> How can I troubleshoot this further ? how can I enable trace >>>>>> logging >>>>>>>> for >>>>>>>>>> management server to see if there is something problematic >>>>>> happening. >>>>>>>>>> >>>>>>>>>> Ammad Ali >>>>>>>>>> >>>>>>>>>> On Sat, Aug 15, 2020 at 1:28 PM Ammad Syed <syedamma...@gmail.com >>>>>>>>>> <mailto:syedamma...@gmail.com>> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I have setup my test environment, here is what I did: >>>>>>>>>>> >>>>>>>>>>> - Installed ACS 4.11.1 and added xenserver 7.0 host in it. >>>>>>> SystemVMs >>>>>>>>> are >>>>>>>>>>> up in the zone and agents are up. >>>>>>>>>>> - Then upgraded the system to 4.11.3. Recreated systemVMs, agent >>>>>> is >>>>>>>> up >>>>>>>>>> and >>>>>>>>>>> systemVM are up with 4.11.3 systemVM version. >>>>>>>>>>> - Then upgraded the system to 4.13.1, recreated systemVMs are >>>>>>> running >>>>>>>>> but >>>>>>>>>>> agents are not up. >>>>>>>>>>> >>>>>>>>>>> I have checked md5sum of systemvm.iso on xenserver and management >>>>>>>>> server, >>>>>>>>>>> both are same. >>>>>>>>>>> >>>>>>>>>>> [root@xenserver iso]# md5sum >>>>>>>> /opt/xensource/packages/iso/systemvm.iso >>>>>>>>>>> baba18f156395da3a5d8208727d8f421 >>>>>>>>>> /opt/xensource/packages/iso/systemvm.iso >>>>>>>>>>> >>>>>>>>>>> [root@cloudstack-upgrade vms]# md5sum >>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso >>>>>>>>>>> baba18f156395da3a5d8208727d8f421 >>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso >>>>>>>>>>> >>>>>>>>>>> Also the private key on xenserver root and management server are >>>>>>>> same. >>>>>>>>>>> >>>>>>>>>>> management server >>>>>>>>>>> /usr/share/cloudstack-common/scripts/vm/systemvm/id_rsa.cloud >>>>>>>>>>> >>>>>>>>>>> xenserver path >>>>>>>>>>> /root/.ssh/id_rsa.cloud >>>>>>>>>>> >>>>>>>>>>> - Ammad Ali >>>>>>>>>>> >>>>>>>>>>> On Thu, Aug 13, 2020 at 11:27 AM Ammad Syed < >>>>>> syedamma...@gmail.com <mailto:syedamma...@gmail.com> >>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Here is the link for download management logs. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> https://drive.google.com/file/d/1l6HDPguGUNaOxsc7VSaj7eaP0F2UOFjA/view?usp=sharing >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Aug 13, 2020 at 11:22 AM Ammad Syed < >>>>>>> syedamma...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I have reverted the version back to 4.11.3. But I have saved >>>>>> logs >>>>>>>>>>>>> starting from upgrade. >>>>>>>>>>>>> >>>>>>>>>>>>> I think the key has been copied successfully in system vm iso. >>>>>>>>>>>>> >>>>>>>>>>>>> 2020-07-25 00:34:17,214 INFO [c.c.s.ConfigurationServerImpl] >>>>>>>>>>>>> (main:null) (logid:) Going to update systemvm iso with >>>>>> generated >>>>>>>>>> keypairs >>>>>>>>>>>>> if needed >>>>>>>>>>>>> 2020-07-25 00:34:17,214 INFO [c.c.s.ConfigurationServerImpl] >>>>>>>>>>>>> (main:null) (logid:) Trying to inject public and private keys >>>>>>> into >>>>>>>>>> systemvm >>>>>>>>>>>>> iso >>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null) >>>>>>> (logid:) >>>>>>>>>>>>> Looking for vms/systemvm.iso in the classpath >>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null) >>>>>>> (logid:) >>>>>>>>>>>>> System resource: >>>>>>> file:/usr/share/cloudstack-common/vms/systemvm.iso >>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null) >>>>>>> (logid:) >>>>>>>>>>>>> Absolute path = /usr/share/cloudstack-common/vms/systemvm.iso >>>>>>>>>>>>> 2020-07-25 00:34:17,218 DEBUG [c.c.s.ConfigurationServerImpl] >>>>>>>>>>>>> (main:null) (logid:) Executing: /bin/bash >>>>>>>>>>>>> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh >>>>>>>>>>>>> /var/cloudstack/management/.ssh/id_rsa.pub >>>>>>>>>>>>> /var/cloudstack/management/.ssh/id_rsa >>>>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso >>>>>>>>>>>>> 2020-07-25 00:34:17,636 INFO [c.c.s.ConfigurationServerImpl] >>>>>>>>>>>>> (main:null) (logid:) Injected public and private keys into >>>>>>> systemvm >>>>>>>>> iso >>>>>>>>>>>>> with result : null >>>>>>>>>>>>> 2020-07-25 00:34:50,613 DEBUG [c.c.h.x.r.CitrixResourceBase] >>>>>>>>>>>>> (DirectAgent-1:ctx-d3dc4cf2) (logid:1d22396d) Copying >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.22 with permission >>>>>>> 0644 >>>>>>>>>>>>> 2020-07-25 00:34:52,566 DEBUG [c.c.h.x.r.CitrixResourceBase] >>>>>>>>>>>>> (DirectAgent-2:ctx-2537b610) (logid:29c67b7a) Copying >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.5 with permission >>>>>> 0644 >>>>>>>>>>>>> 2020-07-25 00:34:53,170 DEBUG [c.c.h.x.r.CitrixResourceBase] >>>>>>>>>>>>> (DirectAgent-3:ctx-168ac27d) (logid:a7862c4b) Copying >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.9 with permission >>>>>> 0644 >>>>>>>>>>>>> 2020-07-25 00:34:54,621 DEBUG [c.c.h.x.r.CitrixResourceBase] >>>>>>>>>>>>> (DirectAgent-6:ctx-f640fe55) (logid:0a62d4cf) Copying >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.5 with permission >>>>>> 0644 >>>>>>>>>>>>> >>>>>>>>>>>>> I have attached complete management logs. The start of logs is >>>>>>> the >>>>>>>>>> start >>>>>>>>>>>>> of management server after upgrade of management server from >>>>>>> 4.11.3 >>>>>>>>> to >>>>>>>>>>>>> 4.13.1. >>>>>>>>>>>>> >>>>>>>>>>>>> Ammad Ali >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Aug 13, 2020 at 1:35 AM Andrija Panic < >>>>>>>>> andrija.pa...@gmail.com >>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Do you get an error while trying to inject ssh key into the >>>>>>>>>> systemvm.iso >>>>>>>>>>>>>> (mgmt logs) , or can you confirm that the systemvm.iso on XS >>>>>>> host >>>>>>>>> and >>>>>>>>>>>>>> the >>>>>>>>>>>>>> one on the mgmt server are identical, md5sum them (i.e. has >>>>>> the >>>>>>>> iso >>>>>>>>>> been >>>>>>>>>>>>>> copied over to the XS successfully) - this might explain not >>>>>>> being >>>>>>>>>> able >>>>>>>>>>>>>> to >>>>>>>>>>>>>> login via ssh with your private key. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best, >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, 12 Aug 2020, 09:08 Ammad Syed, <syedamma...@gmail.com >>>>>>> >>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes this is exactly the same issue that i faced. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Sent from my iPhone >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 12-Aug-2020, at 8:35 AM, Eric Lee Green < >>>>>>>>>>>>>> eric.lee.gr...@gmail.com> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Correct, 4.11.3 template is used for 4.11.3, 4.12, and >>>>>>> 4.13. >>>>>>>>> 4.14 >>>>>>>>>>>>>> moves >>>>>>>>>>>>>>> to the 4.14.0 template. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> There seems to be something odd happening key-wise >>>>>> sometimes >>>>>>>>> with >>>>>>>>>>>>>>> upgrades from 4.11.3 to 4.13.1 or 4.14.0. I managed an >>>>>>> upgrade >>>>>>>>>> from >>>>>>>>>>>>>>> 4.11.3 to 4.13.1 that *almost* worked, but the secondary >>>>>>> storage >>>>>>>>> VM >>>>>>>>>>>>>>> wouldn't work and thus I couldn't spawn new virtual >>>>>> machines. >>>>>>>> Same >>>>>>>>>>>>>> symptom >>>>>>>>>>>>>>> -- key error when the agent tried to ssh into it. And >>>>>> deleting >>>>>>>> it >>>>>>>>>> and >>>>>>>>>>>>>>> making it respawn didn't help. Then I tried 4.11.3 to 4.14.0 >>>>>>> and >>>>>>>>>>>>>> *all* the >>>>>>>>>>>>>>> VM's failed at that point (of course, that was with the new >>>>>>>>>> template). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Right now I'm back at 4.11.3 until this can be figured >>>>>> out. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 8/11/2020 5:53 AM, Ammad Syed wrote: >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I think 4.12 and 4.13 uses same systemVM template i.e >>>>>>> 4.11.3 >>>>>>>>>>>>>> version, >>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>> I already have registered. Currently I am running 4.11.3 >>>>>>>>> version >>>>>>>>>>>>>> of ACS. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> MariaDB [cloud]> SELECT id,name,type,cross_zones,state >>>>>> FROM >>>>>>>>>>>>>>>>> cloud.vm_template WHERE name like '%systemvm-xenserver%' >>>>>>> AND >>>>>>>>>>>>>> removed IS >>>>>>>>>>>>>>>>> NULL; >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>> +------+-----------------------------+--------+-------------+----------+ >>>>>>>>>>>>>>>>> | id | name | type | >>>>>>> cross_zones | >>>>>>>>>>>>>> state | >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>> +------+-----------------------------+--------+-------------+----------+ >>>>>>>>>>>>>>>>> | 337 | systemvm-xenserver-3.0.0 | SYSTEM | >>>>>>> 0 | >>>>>>>>>>>>>> Inactive | >>>>>>>>>>>>>>>>> | 418 | systemvm-xenserver-4.2 | SYSTEM | >>>>>>> 0 | >>>>>>>>>>>>>> Active | >>>>>>>>>>>>>>>>> | 472 | systemvm-xenserver-4.3 | USER | >>>>>>> 1 | >>>>>>>>>>>>>> Inactive | >>>>>>>>>>>>>>>>> | 473 | systemvm-xenserver-4.3 | USER | >>>>>>> 1 | >>>>>>>>>>>>>> Inactive | >>>>>>>>>>>>>>>>> | 474 | systemvm-xenserver-4.3 | USER | >>>>>>> 1 | >>>>>>>>>>>>>> Inactive | >>>>>>>>>>>>>>>>> | 475 | systemvm-xenserver-4.3 | USER | >>>>>>> 1 | >>>>>>>>>>>>>> Inactive | >>>>>>>>>>>>>>>>> | 476 | systemvm-xenserver-4.3 | USER | >>>>>>> 0 | >>>>>>>>>>>>>> Inactive | >>>>>>>>>>>>>>>>> | 479 | systemvm-xenserver-4.3-2 | USER | >>>>>>> 1 | >>>>>>>>>>>>>> Inactive | >>>>>>>>>>>>>>>>> | 480 | systemvm-xenserver-4.3 | SYSTEM | >>>>>>> 0 | >>>>>>>>>>>>>> Active | >>>>>>>>>>>>>>>>> | 549 | systemvm-xenserver-4.5.1 | USER | >>>>>>> 0 | >>>>>>>>>>>>>> Active | >>>>>>>>>>>>>>>>> | 550 | systemvm-xenserver-4.5.1 | SYSTEM | >>>>>>> 0 | >>>>>>>>>>>>>> Active | >>>>>>>>>>>>>>>>> | 651 | systemvm-xenserver-4.7.0 | USER | >>>>>>> 0 | >>>>>>>>>>>>>> Inactive | >>>>>>>>>>>>>>>>> | 652 | systemvm-xenserver-4.7.0 | USER | >>>>>>> 0 | >>>>>>>>>>>>>> Inactive | >>>>>>>>>>>>>>>>> | 653 | systemvm-xenserver-4.7.0 | SYSTEM | >>>>>>> 0 | >>>>>>>>>>>>>> Inactive | >>>>>>>>>>>>>>>>> | 737 | systemvm-xenserver-4.9.2 | SYSTEM | >>>>>>> 1 | >>>>>>>>>>>>>> Inactive | >>>>>>>>>>>>>>>>> | 739 | systemvm-xenserver-4.9.2-sb | SYSTEM | >>>>>>> 1 | >>>>>>>>>>>>>> Active | >>>>>>>>>>>>>>>>> | 1245 | systemvm-xenserver-4.11.1 | SYSTEM | >>>>>>> 1 | >>>>>>>>>>>>>> Active | >>>>>>>>>>>>>>>>> | 1584 | systemvm-xenserver-4.11.2 | SYSTEM | >>>>>>> 1 | >>>>>>>>>>>>>> Active | >>>>>>>>>>>>>>>>> | 1677 | systemvm-xenserver-4.11.3 | SYSTEM | >>>>>>> 1 | >>>>>>>>>>>>>> Active | >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>> +------+-----------------------------+--------+-------------+----------+ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> - Ammad >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Aug 11, 2020 at 5:17 PM Pierre-Luc Dion < >>>>>>>>>>>>>> pdion...@apache.org> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> db. >>>>>>>>>>>>>>>>>> Hi Syed, >>>>>>>>>>>>>>>>>> From 4.12, the systemvm template had to be upgraded >>>>>>> because >>>>>>>> of >>>>>>>>>> OS >>>>>>>>>>>>>>> change in >>>>>>>>>>>>>>>>>> the template, moved to a latest version of Debian. >>>>>> Because >>>>>>>> of >>>>>>>>>>>>>> that, >>>>>>>>>>>>>>> some VR >>>>>>>>>>>>>>>>>> scripts have changed and make obsolete older version of >>>>>>> VRs, >>>>>>>>> so >>>>>>>>>>>>>> you >>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>> most likely have to register an updated systemvm >>>>>> templates >>>>>>>> and >>>>>>>>>>>>>> upgrade >>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>>> system VMs and VRs. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Tue, Aug 11, 2020 at 6:24 AM Ammad Syed < >>>>>>>>>>>>>> syedamma...@gmail.com> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Guys, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I was previously on 4.9.3 cloudstack and upgraded to >>>>>>> 4.11.1 >>>>>>>>>> then >>>>>>>>>>>>>>> 4.11.3. >>>>>>>>>>>>>>>>>>> The version 4.11.3 is working fine since six months. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Now I have tried to upgrade my system from 4.11.3 to >>>>>>>> 4.13.1. >>>>>>>>>> The >>>>>>>>>>>>>>> upgrade >>>>>>>>>>>>>>>>>>> goes successful. I didn't uploaded any system VM >>>>>>> template. >>>>>>>>>>>>>> However the >>>>>>>>>>>>>>>>>>> problem occured when I recreated my systemVM of POD, >>>>>> the >>>>>>> VM >>>>>>>>>>>>>> recreated >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> its state was running but agent state was not getting >>>>>> up, >>>>>>>> its >>>>>>>>>>>>>> showing >>>>>>>>>>>>>>>>>> blank >>>>>>>>>>>>>>>>>>> in column. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Digging further via job logs, the job is failed with >>>>>>> error >>>>>>>>> that >>>>>>>>>>>>>>> unable to >>>>>>>>>>>>>>>>>>> execute command via ssh. Below are the logs. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,126 ERROR [c.c.u.s.SshHelper] >>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) SSH >>>>>>>> execution >>>>>>>>>> of >>>>>>>>>>>>>>> command >>>>>>>>>>>>>>>>>>> /opt/cloud/bin/router_proxy.sh keystore-s >>>>>>>>>>>>>>>>>>> etup 169.254.2.199 >>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/agent.properties >>>>>>>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/cloud.jks >>>>>> TJaQYChYBwKh7Cx9 >>>>>>>> 365 >>>>>>>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/clou >>>>>>>>>>>>>>>>>>> d.csr has an error status code in return. Result >>>>>> output: >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG >>>>>>> [c.c.a.m.DirectAgentAttache] >>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq >>>>>>>>>>>>>>>>>>> 906-3195585410596077730: Response Received: >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request] >>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq >>>>>>>>>>>>>>>>>>> 906-3195585410596077730: Processing: { Ans: , MgmtId: >>>>>>>>>> 779271079 >>>>>>>>>>>>>>>>>>> 43497, via: 906(xen-21-10-a3-khi02), Ver: v1, Flags: >>>>>> 10, >>>>>>>>>>>>>>>>>>> [{"org.apache.cloudstack.ca >>>>>>>>>>>>>>>>>>> .SetupKeystoreAnswer":{"result":false,"wait":0}}] >>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request] >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d >>>>>>> job-1208155/job-1208258 >>>>>>>>>>>>>>> ctx-df740f75) >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Seq 906-319558541059607773 >>>>>>>>>>>>>>>>>>> 0: Received: { Ans: , MgmtId: 77927107943497, via: >>>>>>>>>>>>>>>>>>> 906(xen-21-10-a3-khi02), Ver: v1, Flags: 10, { >>>>>>>>>>>>>> SetupKeystoreAnswer } } >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 ERROR >>>>>>>>> [c.c.v.VirtualMachineManagerImpl] >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d >>>>>>> job-1208155/job-1208258 >>>>>>>>>>>>>>> ctx-df740f75) >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Failed to setup keystore and generate >>>>>>> CSR >>>>>>>>> for >>>>>>>>>>>>>> system >>>>>>>>>>>>>>> vm: >>>>>>>>>>>>>>>>>>> s-24142-VM >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG >>>>>>> [c.c.v.VmWorkJobHandlerProxy] >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d >>>>>>> job-1208155/job-1208258 >>>>>>>>>>>>>>> ctx-df740f75) >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Done executing VM work job: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":24142,"handlerName":"VirtualMachineManagerImpl"} >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,128 DEBUG >>>>>>>>> [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d >>>>>>> job-1208155/job-1208258 >>>>>>>>>>>>>>> ctx-df740f75) >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Complete async job-1208258, jobStatus: >>>>>>>>>>>>>> SUCCEEDED, >>>>>>>>>>>>>>>>>>> resultCode: 0, result: null >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I tried to dig it further, I was unable to login >>>>>> systemVM >>>>>>>> via >>>>>>>>>>>>>> ssh from >>>>>>>>>>>>>>>>>>> xenserver host with key /root/.ssh/id_rsa.cloud placed. >>>>>>>> Look >>>>>>>>>> like >>>>>>>>>>>>>>> private >>>>>>>>>>>>>>>>>>> key issue. However I am able to login on my old >>>>>>> systemVMs ( >>>>>>>>> i.e >>>>>>>>>>>>>>> created >>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>> ACS 4.11.3) >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Also I have SSL certificate enabled for console proxy >>>>>> on >>>>>>> my >>>>>>>>> ACS >>>>>>>>>>>>>> 4.11.3 >>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> I am using only xenserver 7.0 hosts. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I tried to disable SSL on secstorage and console proxy >>>>>>> from >>>>>>>>>>>>>> global >>>>>>>>>>>>>>>>>>> settings, but still didn't worked. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I had a fresh installation of ACS 4.13.1 with xenserver >>>>>>>> 7.0, >>>>>>>>>>>>>> systemVMs >>>>>>>>>>>>>>>>>> are >>>>>>>>>>>>>>>>>>> working fine in it. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Please advise. >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Syed Ammad Ali >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Syed Ammad Ali >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Regards, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Syed Ammad Ali >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Syed Ammad Ali >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Syed Ammad Ali >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Andrija Panić >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> >>>>>>>> >>>>>>>> Syed Ammad Ali >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Andrija Panić >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> >>>>>> >>>>>> Syed Ammad Ali >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Andrija Panić >>>> >>>> >>>> -- >>>> Regards, >>>> >>>> >>>> Syed Ammad Ali >> >
