Hi all, We are running 4.11.3 with a single zone, that zone is working without issue. We are trying to add a second zone to the installation, and everything seems to go well, except we are confused on how the SystemVM templates should be handled for the new zone. The new zone has its own secondary storage (NFS). When Cloudstack sees the new Zone, it attempts to provision a Secondary Storage VM. However, it is unable to because the SystemVM Template doesn't exist on the new secondary storage (NFS).
Are we supposed to pre-populate another copy of the SystemVM Template in the additional zone and secondary storage? Or should cloudstack copy the existing SystemVM Template (which is set as cross-zone) to the new zone for us? Here is some detailed information: MariaDB [cloud]> SELECT id,name,type,cross_zones,state FROM cloud.vm_template WHERE name like '%systemvm-kvm%' AND removed IS NULL; +-----+------------------------------+---------+-------------+----------+ | id | name | type | cross_zones | state | +-----+------------------------------+---------+-------------+----------+ | 344 | systemvm-kvm-4.11.3 | SYSTEM | 1 | Active | +-----+------------------------------+---------+-------------+----------+ MariaDB [cloud]> select id,store_id,template_id,install_path, download_state from template_store_ref; +-----+----------+-------------+------------------------------------------------------------------------+----------------+ | id | store_id | template_id | install_path | download_state | +-----+----------+-------------+------------------------------------------------------------------------+----------------+ | 666 | 1 | 344 | template/tmpl/2/344/182f0a79-1e16-3e53-a6e9-fcffe5f11c3e.qcow2 | DOWNLOADED | | 756 | 16 | 344 | template/tmpl/1/344/ | DOWNLOADED | +-----+----------+-------------+------------------------------------------------------------------------+----------------+ Why in the template_store_ref table did cloudstack add a new row with "downloaded" and missing a filename in the "install_path"? The KVM host cannot mount the template on the new secondary storage, because it isn't there yet (should cloudstack be copying that template from the existing zone to the new one for us?): ---------------------- 2020-03-27 18:51:40,626 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:6b50f03a) Failed to create netfs mount: 10.102.33.5:/zone2_secondary/template/tmpl/1/344 org.libvirt.LibvirtException: internal error: Child process (/usr/bin/mount 10.10.33.5:/zone2_secondary/template/tmpl/1/344 /mnt/b69caab0-4c1e-34b6-94b8-2617ba561e9a -o nodev,nosuid,noexec) unexpected exit status 32: mount.nfs: mounting 10.10.33.5:/zone2__secondary/template/tmpl/1/344 failed, reason given by server: No such file or directory --------------------- Thanks! Sean