Hi Will, Afaik the deleteTemplate API removes the template in CloudStack (database entries) but does not remove the template files on storage. My statement was a concern about replacing the template files which can be used as a base image when using thin deployment on VMs, which can affect the running VMs.
About the template versions, I think currently the only way to achieve this is to register a new template for a newer version. About the prepareTemplate API, I don’t think that will achieve the goal but could require some testing Regards, Nicolas Vazquez From: Will Conrad <wcon...@hivelocity.net.INVALID> Date: Tuesday, 20 June 2023 at 15:58 To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: Direct Download/Bypass Secondary Storage option for templates Can you elaborate on this statement? "but only if there aren’t VMs using the template as the backing file on that primary storage pool." The documentation here states that deleting a template does not affect any VMs using the template. https://cloudstack.apache.org/api/apidocs-4.18/apis/deleteTemplate.html Does cloudstack have a recommended best practice for managing templates/versions? Say, for instance, If we want to ensure we have the latest cloud image every two weeks for an ubuntu2204 template? Would manually deleting the cached file from primary storage and running the prepareTemplate API call on the template do what I seek?Would it be safe? prepareTemplate API documented here: https://cloudstack.apache.org/api/apidocs-4.18/apis/prepareTemplate.html On Tue, Jun 20, 2023 at 10:51 AM Nicolas Vazquez < nicolas.vazq...@shapeblue.com> wrote: > Hi Will, > > You can refer to the Github repository > https://github.com/apache/cloudstack and submit a PR for this, or can > also raise an issue and describe this functionality as a new > feature/improvement: https://github.com/apache/cloudstack/issues/new. > > The only problem I see with your approach is that only updating secondary > storage will not ensure that new dpeloyments will get the latest version, > since CloudStack keeps copies of the template on primary storage also. New > deployments use the cached templates on primary storage when it is > available to avoid copying from secondary storage. I think that the feature > should remove all the existing copies in primary storage also and replace > them with the latest version, but only if there aren’t VMs using the > template as the backing file on that primary storage pool. > > Regards, > Nicolas Vazquez > > > From: Will Conrad <wcon...@hivelocity.net.INVALID> > Date: Tuesday, 20 June 2023 at 08:46 > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > Subject: Re: Direct Download/Bypass Secondary Storage option for templates > Hi Nicolas, > > Where do we stand on this? Is this something that could be easily updated > or feature added? I'm not familiar with the code base. If I were to explore > the idea of making an update and submitting a PR, where would I start? > > Regards, > > Willard > > On Fri, Jun 16, 2023 at 6:41 AM Will Conrad <wcon...@hivelocity.net> > wrote: > > > Nicolas, > > > > "In your requirement do you mean to change the template URL and > > re-download the template from a different location or simply download > again > > the template from the same URL after updating the served file?" > > > > For our use case the URL will stay the same, but the file will be updated > > (though I could imagine other use cases where an update of the URL would > be > > useful too). Consider a URL like > > http://some.distro.com/cloud/latest/imgname.img > > where /latest/ automatically takes you to the latest updated version of > > that file over time. We would not be looking to trigger an update on all > of > > the primary storage pools where instances have been deployed, just the > > secondary storage such that new deploys use the updated image. > > > > Over time, distro cloud images get updated with the latest updates (that > > one would receive just by running apt update && apt upgrade, for > instance). > > If we continue to deploy with the same image, updates will cause vm > > deployment to take longer and longer over time, as well as consume more > and > > more network bandwidth, due to the updates each vm will have to perform > at > > time of deployment. > > > > "Afaik there is no supported way to safely do this except manually > > updating the files on the downloaded storage pools (secondary and > primary) > > which should be updated carefully. The tables template_store_ref and > > template_spool_ref indicate the downloaded copies on secondary and > primary > > pools of each template." > > > > I assume you're referring to the question about copying over/updating the > > secondary storage image manually. I consider this idea a hack that could > > cause issues or at the very least be at risk of breaking when a new > > cloudstack update gets deployed. I don't like the idea myself. I'm just > > trying to explore all the options available to us and see where your > > guidance takes me. > > > > The reality is that we can make direct download work, but to do so would > > require additional infrastructure (for instance, a webserver in each > zone > > where we point the URL to, that we now have to maintain and monitor). > > Ideally we would just like to let the secondary storage server(s) fulfill > > their role. > > > > On Thu, Jun 15, 2023 at 10:30 PM Nicolas Vazquez < > > nicolas.vazq...@shapeblue.com> wrote: > > > >> Hi Will, > >> > >> When registering a template CloudStack usually downloads it first to > >> secondary storage and then copies it through the different primary > storage > >> pools when a VM deployment requires it. The aim of the direct download > >> feature is to skip the first step and directly downloading the templates > >> into primary storage pools without any secondary storage intervention. > >> > >> In your requirement do you mean to change the template URL and > >> re-download the template from a different location or simply download > again > >> the template from the same URL after updating the served file? Afaik > there > >> is no supported way to safely do this except manually updating the > files on > >> the downloaded storage pools (secondary and primary) which should be > >> updated carefully. The tables template_store_ref and template_spool_ref > >> indicate the downloaded copies on secondary and primary pools of each > >> template. > >> > >> Regards, > >> Nicolas Vazquez > >> > >> > >> From: Will Conrad <wcon...@hivelocity.net.INVALID> > >> Date: Thursday, 15 June 2023 at 14:47 > >> To: users@cloudstack.apache.org <users@cloudstack.apache.org> > >> Subject: Re: Direct Download/Bypass Secondary Storage option for > templates > >> Nicolas, > >> > >> The reason we're considering using the directdownload feature is to > >> simplify template maintenance/updates. I presume that's what it was > >> designed for. We want to be able to, preferably through cloudstack > >> functionality, update the template image file associated with a > template. > >> We planned on achieving this utilizing directdownload to decouple the > >> image > >> file from the registered template itself when it occurred to us that a > >> "regrab" button in the template properties webui or an API call to tell > >> secondary storage to redownload the source would very much simplify this > >> process. > >> > >> This brings my questions to: > >> > >> How difficult would it be to implement something like that? > >> > >> Is there another way to update the imagefile associated with a > template? I > >> mean, could we manually overwrite the file on secondary storage? Would > >> that > >> break anything? > >> > >> What is Cloudstack's recommended best practice for managing template > >> images? > >> > >> > >> Regards, > >> > >> Willard (Will) > >> > >> > >> > >> On Wed, Jun 14, 2023 at 10:26 AM Nicolas Vazquez < > >> nicolas.vazq...@shapeblue.com> wrote: > >> > >> > No problem, I think these docs do not clearly state the supported > >> storage > >> > providers, I will fix that. On this blog entry we have mentioned them: > >> > > >> > https://www.shapeblue.com/cloudstack-feature-first-look-direct-download-agnostic-of-the-storage-provider/ > >> > > >> > Currently the direct download feature is supported on NFS, local > storage > >> > and shared mount point, but not for Ceph. > >> > > >> > Regards, > >> > Nicolas Vazquez > >> > > >> > > >> > From: Will Conrad <wcon...@hivelocity.net.INVALID> > >> > Date: Wednesday, 14 June 2023 at 10:58 > >> > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > >> > Subject: Re: Direct Download/Bypass Secondary Storage option for > >> templates > >> > Nicolas, > >> > > >> > I feel silly for not having read that documentation all the way > through. > >> > Thank you for your assistance. > >> > > >> > I have another question, now. Since we've been working with this we > >> > have been trying various methods of testing directdownload templates. > >> Since > >> > we were having problems with HTTPS, we tested HTTP. We have run into a > >> > problem where the template fails to download if the guest is using > ceph > >> > storage. When we change to creating the VM on "local" storage, the > >> template > >> > download succeeds and the VM creates successfully. Are there any > >> insights > >> > you can provide here? Is there more documentation I may have missed? > >> > > >> > On Wed, Jun 14, 2023 at 9:39 AM Nicolas Vazquez < > >> > nicolas.vazq...@shapeblue.com> wrote: > >> > > >> > > Thanks Will, > >> > > > >> > > Currently it is only possible to upload the certificate via API but > >> not > >> > > from the UI, please find it documented here: > >> > > > >> > > >> > https://docs.cloudstack.apache.org/en/latest/adminguide/templates.html#bypassing-secondary-storage-for-kvm-templates > >> > > . > >> > > > >> > > In your case as the template is stored on Github you may want to > >> upload a > >> > > Github certificate to the hosts for the download to be trusted > >> > > > >> > > Regards, > >> > > Nicolas Vazquez > >> > > > >> > > > >> > > From: Will Conrad <wcon...@hivelocity.net.INVALID> > >> > > Date: Wednesday, 14 June 2023 at 10:06 > >> > > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > >> > > Subject: Re: Direct Download/Bypass Secondary Storage option for > >> > templates > >> > > Hi Wei and Nicolas, > >> > > > >> > > Thank you for you responses. > >> > > > >> > > Wei, > >> > > > >> > > I checked the host, and confirmed that yes the ca-certificates > >> package is > >> > > installed and latest. > >> > > "root@lax2-cs-hv01:~# apt list ca-certificates -a > >> > > > >> > > Listing... Done > >> > > > >> > > ca-certificates/jammy-updates,jammy-security,now > >> 20230311ubuntu0.22.04.1 > >> > > all [installed,automatic] > >> > > > >> > > ca-certificates/jammy 20211016 all > >> > > > >> > > > >> > > > >> > > Nicolas, > >> > > > >> > > "Have you tried uploading the required certificate for the https > >> download > >> > > via the uploadTemplateDirectDownloadCertificate API?" > >> > > > >> > > No I have not. I was unaware of the need to do this. Is there > >> > documentation > >> > > I may have missed? What certificate is supposed to be uploaded and > >> how is > >> > > it used? > >> > > > >> > > Regards, > >> > > > >> > > Willard > >> > > > >> > > On Tue, Jun 13, 2023 at 10:01 PM Nicolas Vazquez < > >> > > nicolas.vazq...@shapeblue.com> wrote: > >> > > > >> > > > Hi Will, > >> > > > > >> > > > Have you tried uploading the required certificate for the https > >> > download > >> > > > via the uploadTemplateDirectDownloadCertificate API? > >> > > > > >> > > > Regards, > >> > > > Nicolas Vazquez > >> > > > > >> > > > > >> > > > From: Wei ZHOU <ustcweiz...@gmail.com> > >> > > > Date: Tuesday, 13 June 2023 at 20:01 > >> > > > To: users@cloudstack.apache.org <users@cloudstack.apache.org> > >> > > > Subject: Re: Direct Download/Bypass Secondary Storage option for > >> > > templates > >> > > > Hi Will, > >> > > > > >> > > > What hypervisor do you use ? Have you installed ca-crrtificates > >> > package? > >> > > > > >> > > > -Wei > >> > > > > >> > > > On Tuesday, 13 June 2023, Will Conrad <wcon...@hivelocity.net > >> .invalid> > >> > > > wrote: > >> > > > > >> > > > > Hello again, Community! > >> > > > > > >> > > > > We're trying to make use of DirectDownload templates which makes > >> use > >> > of > >> > > > the > >> > > > > "Bypass Secondary Storage" feature, but we seem to be having > >> issues > >> > > with > >> > > > > this functionality. > >> > > > > > >> > > > > After setting up a new template with "Direct Download" turned on > >> and > >> > an > >> > > > > HTTPS URL our template file won't download. The download source > >> is a > >> > > file > >> > > > > stored in github. This is what we see in the logs: > >> > > > > > >> > > > > Jun 13 16:12:07 lax2-cs-hv01 java[26054]: WARN > >> > > > > [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-5:) > >> > > > > (logid:7b08521c) Error downloading template 209 due to: Error on > >> > HTTPS > >> > > > > request: PKIX path building failed: > >> > > > > sun.security.provider.certpath.SunCertPathBuilderException: > >> unable to > >> > > > > find valid certification path to requested target > >> > > > > Jun 13 16:12:07 lax2-cs-hv01 java[26054]: INFO > >> > > > > [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:) > >> > > > > (logid:7b08521c) Trying to fetch storage pool > >> > > > > 3b59a095-9e71-3e97-92a8-56aa3f931a5e from libvirt > >> > > > > Jun 13 16:12:07 lax2-cs-hv01 java[26054]: WARN > >> > > > > [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-1:) > >> > > > > (logid:7b08521c) Error downloading template 209 due to: Error on > >> > HTTPS > >> > > > > request: PKIX path building failed: > >> > > > > sun.security.provider.certpath.SunCertPathBuilderException: > >> unable to > >> > > > > find valid certification path to requested target > >> > > > > Jun 13 16:12:07 lax2-cs-hv01 java[26054]: INFO > >> > > > > [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:) > >> > > > > (logid:78a6fa93) Trying to fetch storage pool > >> > > > > 3b59a095-9e71-3e97-92a8-56aa3f931a5e from libvirt > >> > > > > Jun 13 16:12:07 lax2-cs-hv01 java[26054]: INFO > >> > > > > [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:) > >> > > > > (logid:78a6fa93) Asking libvirt to refresh storage pool > >> > > > > 3b59a095-9e71-3e97-92a8-56aa3f931a5e > >> > > > > Jun 13 16:12:07 lax2-cs-hv01 java[26054]: INFO > >> > > > > [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:) > >> > > > > (logid:7b08521c) Trying to fetch storage pool > >> > > > > 3b59a095-9e71-3e97-92a8-56aa3f931a5e from libvirt > >> > > > > Jun 13 16:12:08 lax2-cs-hv01 java[26054]: INFO > >> > > > > [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:) > >> > > > > (logid:78a6fa93) Trying to fetch storage pool > >> > > > > eb9f16ef-3ba3-4c50-9e64-807b6f2c8994 from libvirt > >> > > > > Jun 13 16:12:08 lax2-cs-hv01 java[26054]: INFO > >> > > > > [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:) > >> > > > > (logid:78a6fa93) Asking libvirt to refresh storage pool > >> > > > > eb9f16ef-3ba3-4c50-9e64-807b6f2c8994 > >> > > > > Jun 13 16:12:08 lax2-cs-hv01 java[26054]: WARN > >> > > > > [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-2:) > >> > > > > (logid:7b08521c) Error downloading template 209 due to: Error on > >> > HTTPS > >> > > > > request: PKIX path building failed: > >> > > > > sun.security.provider.certpath.SunCertPathBuilderException: > >> unable to > >> > > > > find valid certification path to requested target > >> > > > > > >> > > > > We've been through this documentation: > >> > > > > > >> https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html#< > >> > > > > https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html > >> ><<< > >> > https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html%3e%3c%3c > >> > > >> > > >> > https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html%3e%3c > >> >< > >> > https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html%3e%3c%3e > >> > > >> > > > https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html%3e > >> ><< > >> > https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html%3e%3e%3c > >> > > >> > > >> > https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html%3e%3e > >> >< > >> > https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html%3e%3e%3e > >> > > >> > > > > securing-process > >> > > > > > >> > > > > but everything seems to be in order, on our side. Any insights > >> here? > >> > > > > Happy to provide any logs or configuration information to > assist. > >> > > > > > >> > > > > Regards, > >> > > > > > >> > > > > Willard Conrad > >> > > > > > >> > > > > DevOps Engineer > >> > > > > > >> > > > > Hivelocity, LLC > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > >> > > > >> > > > >> > > >> > > >> > > >> > > >> > >> > >> > >> > > > >
