I posted a couple weeks back about some issues getting ISO's and templates uploaded in ACS and received some excellent help. I've changed a few things around based on that previous issue and am finally circling back to it but still running into a problem uploading an ISO. When I try to upload an ISO through primate I now get two messages [1]:
Network Error: Unable to reach the management server or a browser extension may be blocking the network request. Upload Failed: Failed to upload ISO - Error: Network Error After I click "OK" on the "Upload ISO from Local" dialog box I get these two messages/errors and nothing else happens, the dialog box doesn't go away and I can click "OK" again as many times as I want (the messages always repeat). Also, If I refresh the ISO page then I see that my ISO that I tried to upload is now listed [2], but is not actually ready [3]. To give some background, I am running ACS 4.15.0.0 with two load balanced management servers and a KVM compute host. I've tried the following: 1. Disabled one of the management servers. 2. Connected directly to a management server. 3. Used different browsers. Nothing listed above has changed the outcome and the same message is always thrown, so I don't think it is a load balancer or browser issue. As this is related to my previous issue in getting the ssvm started I wanted to show that it is now up and running [4], I can connect to it, and the /usr/local/cloud/systemvm/ssvm-check.sh script doesn't throw any errors: root@s-41-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh ================================================ First DNS server is 10.2.81.1 PING 10.2.81.1 (10.2.81.1): 56 data bytes 64 bytes from 10.2.81.1: icmp_seq=0 ttl=63 time=0.357 ms 64 bytes from 10.2.81.1: icmp_seq=1 ttl=63 time=0.399 ms --- 10.2.81.1 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.357/0.378/0.399/0.000 ms Good: Can ping DNS server ================================================ Good: DNS resolves cloudstack.apache.org ================================================ nfs is currently mounted Mount point is /mnt/SecStorage/df2ca46d-aee0-302e-9ad0-2e94252341e4 Good: Can write to mount point ================================================ Management server is bllcloudlb01.harmonywave.cloud. Checking connectivity. Good: Can connect to management server bllcloudlb01.harmonywave.cloud port 8250 ================================================ Good: Java process is running ================================================ Tests Complete. Look for ERROR or WARNING above. The cloud service inside the system VM is also running: root@s-41-VM:~# service cloud status ● cloud.service - CloudStack Agent service Loaded: loaded (/etc/systemd/system/cloud.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2021-06-26 02:58:24 UTC; 6 days ago Main PID: 3011 (bash) Tasks: 48 (limit: 543) Memory: 157.0M CGroup: /system.slice/cloud.service ├─3011 bash /usr/local/cloud/systemvm/_run.sh └─3241 java -Djavax.net.ssl.trustStore=./certs/realhostip.keystore -Djdk.tls.ephemeralDHKeySize=2048 -Djsse.enableSNIExtension=false -Dlog.hom Jul 02 15:16:41 s-41-VM _run.sh[3011]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) Jul 02 15:16:41 s-41-VM _run.sh[3011]: at java.base/java.lang.Thread.run(Thread.java:834) Jul 02 15:16:41 s-41-VM _run.sh[3011]: 15:16:41,284 INFO Agent:835 - Processing agent ready command, agent id = 16 Jul 02 15:16:41 s-41-VM _run.sh[3011]: 15:16:41,284 INFO Agent:375 - Set agent id 16 Jul 02 15:16:41 s-41-VM _run.sh[3011]: 15:16:41,285 INFO Agent:842 - Ready command is processed for agent id = 16 Jul 02 15:16:41 s-41-VM _run.sh[3011]: 15:16:41,459 INFO Agent:835 - Processing agent ready command, agent id = 16 Jul 02 15:16:41 s-41-VM _run.sh[3011]: 15:16:41,459 INFO Agent:375 - Set agent id 16 Jul 02 15:16:41 s-41-VM _run.sh[3011]: 15:16:41,469 INFO Agent:791 - Processed new management server list: bllcloudlb01.harmonywave.cloud@static Jul 02 15:16:41 s-41-VM _run.sh[3011]: 15:16:41,469 INFO Agent:842 - Ready command is processed for agent id = 16 Jul 02 15:16:45 s-41-VM _run.sh[3011]: 15:16:45,134 INFO Agent:549 - Connected to the host: bllcloudlb01.harmonywave.cloud Finally I took a network capture from the browser when I hit "OK" and I see the following: [5] https://drive.google.com/file/d/1w-gnARJ1XegK5D24c91T3qseU8GDNAPd/view?usp=sharing I'm not quite sure what to make of that and can provide more details if needed. I have been able to verify that port 443 is up on the SSVM: root@s-41-VM:~# ss -tupanl | grep 443 tcp LISTEN 0 511 192.41.41.161:443 0.0.0.0:* users:(("apache2",pid=28140,fd=4),("apache2",pid=28139,fd=4),("apache2",pid=28136,fd=4)) The logs don't seem to provide me with any hint of the issue: ################### # MANAGEMENT LOGS # ################### 2021-07-02 14:37:39,767 DEBUG [c.c.a.ApiServlet] (qtp1644231115-26571:ctx-3657db16) (logid:2a9a3fec) ===START=== 172.16.44.18 -- GET name=ubuntu-20.04.2-live-server-amd64.iso&displaytext=Ubuntu+20.04+(&zoneid=9f6f5b49-0e12-4af4-a13b-2ace6c47de43&bootable=true&ostypeid=aa18f3ad-cd73-11eb-b1da-5254008f72d5&isextractable=false&ispublic=true&isfeatured=false&format=ISO&command=getUploadParamsForIso&response=json 2021-07-02 14:37:39,787 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-11:null) (logid:) SeqA 15-71382: Processing Seq 15-71382: { Cmd , MgmtId: -1, via: 15, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"40","_loadInfo":"{ "connections": [] }","wait":"0"}}] } 2021-07-02 14:37:39,805 DEBUG [c.c.a.ApiServer] (qtp1644231115-26571:ctx-3657db16 ctx-d1285d49) (logid:2a9a3fec) CIDRs from which account 'Acct[f8d6949d-cd74-11eb-b1da-5254008f72d5-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0 2021-07-02 14:37:39,826 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-11:null) (logid:) SeqA 15-71382: Sending Seq 15-71382: { Ans: , MgmtId: 90520733511963, via: 15, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0"}}] } 2021-07-02 14:37:40,000 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (qtp1644231115-26571:ctx-3657db16 ctx-d1285d49) (logid:2a9a3fec) template 217 is not in store:2, type:Image 2021-07-02 14:37:40,032 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (qtp1644231115-26571:ctx-3657db16 ctx-d1285d49) (logid:2a9a3fec) template 217 is already in store:2, type:Image 2021-07-02 14:37:40,058 DEBUG [c.c.r.ResourceLimitManagerImpl] (qtp1644231115-26571:ctx-3657db16 ctx-d1285d49) (logid:2a9a3fec) Updating resource Type = template count for Account = 2 Operation = increasing Amount = 1 2021-07-02 14:37:40,289 DEBUG [c.c.a.ApiServlet] (qtp1644231115-26571:ctx-3657db16 ctx-d1285d49) (logid:2a9a3fec) ===END=== 172.16.44.18 -- GET name=ubuntu-20.04.2-live-server-amd64.iso&displaytext=Ubuntu+20.04+(&zoneid=9f6f5b49-0e12-4af4-a13b-2ace6c47de43&bootable=true&ostypeid=aa18f3ad-cd73-11eb-b1da-5254008f72d5&isextractable=false&ispublic=true&isfeatured=false&format=ISO&command=getUploadParamsForIso&response=json 2021-07-02 14:37:40,496 DEBUG [c.c.a.ApiServlet] (qtp1644231115-10794:ctx-8910f7db) (logid:debd3cae) ===START=== 172.16.44.18 -- GET listall=true&isofilter=all&showunique=true&page=1&pagesize=10&command=listIsos&response=json ################# # LOAD BALANCER # ################# Jul 2 14:37:40 bllcloudlb01 haproxy[216322]: 172.16.44.18:54094 [02/Jul/2021:14:37:39.761] ft_cloudstack_gui~ bk_cloudstack_gui/cloudstack_gui_1 1/0/0/530/531 200 1131 - - --VN 4/1/0/0/0 0/0 "GET /client/api/?name=ubuntu-20.04.2-live-server-amd64.iso&displaytext=Ubuntu+20.04+(&zoneid=9f6f5b49-0e12-4af4-a13b-2ace6c47de43&bootable=true&ostypeid=aa18f3ad-cd73-11eb-b1da-5254008f72d5&isextractable=false&ispublic=true&isfeatured=false&format=ISO&command=getUploadParamsForIso&response=json HTTP/1.1" Jul 2 14:37:40 bllcloudlb01 haproxy[216322]: 172.16.44.18:54094 [02/Jul/2021:14:37:40.491] ft_cloudstack_gui~ bk_cloudstack_gui/cloudstack_gui_1 0/0/0/164/164 200 994 - - --VN 4/1/0/0/0 0/0 "GET /client/api/?listall=true&isofilter=all&showunique=true&page=1&pagesize=10&command=listIsos&response=json HTTP/1.1" Any help would be appreciated. [1] (https://drive.google.com/file/d/1cIZtMXfpy83etjq8NGdo-x8UVcPDiZZw/view?usp=sharing) [2] (https://drive.google.com/file/d/1IHDmteKXFjKli0u7rzFQVznIjSw8chD7/view?usp=sharing) [3] (https://drive.google.com/file/d/1FWyR4Dzo20UNDO887V6tpyIej5o0FRUd/view?usp=sharing) [4] (https://drive.google.com/file/d/1N1dTIYvxIjTSgSaf56SCmoQlQNou7vNW/view?usp=sharing) [5] (https://drive.google.com/file/d/1w-gnARJ1XegK5D24c91T3qseU8GDNAPd/view?usp=sharing) -- Thanks, Joshua Schaeffer
