Re: KVM VM does not allow to take a disk-only snapshot when VM is in running state
Hello, As Wei said, for VM snapshots, only disk+memory is supported for running VMs. However, ACS + KVM allow operators to take disk-only snapshots of specific volumes; for that, the global setting 'KVM.snapshot.enabled' must be 'true'. After changing it to 'true', one must logout and login to the platform again (to refresh this data); then, when one navigates to a specific volume, the button to take a volume snapshot should appear, whether the VM is running or not. Could you verify the global setting 'KVM.snapshot.enabled' and try this approach? Best regards, Daniel Salvador (gutoveronezi) On 20/01/2023 07:21, Wei ZHOU wrote: Hi, For VMs on kvm, only disk+memory vmsnapshot is possible for running vms. Please tick the "Snapshot memory" when you create a vm snapshot. When vm is stopped, you can create a volume snapshot (not vm snapshot). -Wei On Fri, 20 Jan 2023 at 09:47, I Made Ocy Darma Putra < ocy.da...@indochat.co.id> wrote: I think i have problem that i can’t create snapshot when it is in running state, but when i stop the VM, the Snapshot buttons are dissapear
Re: [4.18][RELEASE] timeline
Hello Daan, The timeline LGTM. Best regards, Daniel Salvador (gutoveronezi) On 03/01/2023 06:39, Daan Hoogland wrote: Happy new year to all, I wish you enjoyment wisdom health and strength in your work and your private lives. I wish to inform you of my plans for the coming weeks as a Release Manager: At the moment we have - 32 issues and - 31 pull requests marked for 4.18. These numbers were a bit lower end of last year and to prevent shooting at a moving target I want to share the following. This week, I want to create a milestone 4.18.1.0 and move any issue that - reads as a bug report - does not have someone assigned - does not have a linked pull request In addition I want to move to milestone 4.19.0.0, any issue that - reads as an enhancement or a feature request - does not have a pull request linked In two weeks, on the 17th, I want to call a freeze on merges to main for stabilization. At the end of January, approximately on the 31st I want to release RC1. I am open to any request for change in this schedule. There are some hefty PRs out that I imagine everybody wants in, and you may have quick and handy or vital fixes to add as well. Please feel free to address me on this thread or on any DM I appear responsive on. regards,
Re: How to dedicate host or cluster to projects?
Hello, Jorge Via UI it is not possible; however, the API for dedicating resources receives the parameters account and domainid. ACS creates an account for every project created (as found out). Therefore, you can call the APIs passing the name of the account created by ACS (the one with prefix 'PrjAcct-') and the domain related to the project. I did a few tests in our QA and it worked fine (dedicated a host to a project and created a VM for it - outside the project it was not possible to instantiate a VM to that host); however, deeper tests should be done before using this approach in production. Best regards, Daniel Salvador (gutoveronezi) On 04/10/2022 09:39, Jorge Luiz Correa wrote: Looking at documentation I can't find a way to dedicate a host or cluster to a project. Is there some way to do that today? Cloudstack intends to make it possible some day? Each project creates an account and we can dedicate hosts and clusters to accounts. So, it looks like to be the same thing, right? Thank you!
Re: IP address to use for consoleproxy.url.domain
Hallo Aufgabe,
We experienced a similar situation while deploying our testing laboratory (QA).
The configuration 'consoleproxy.url.domain' must be a value that would allow
the console to connect with the Management Servers. As we did not want a domain
for our QA and would just access it via IP, we set the MS IP as the
configuration value. However, for this configuration ACS requires a value that
matches a domain ('something.any'); therefore, to set the IP to the
configuration you will have to manually intervene in the database:
UPDATE cloud.configuration
SET value = ''
WHERE name = 'consoleproxy.url.domain';
If you have a load balancer configured, you would also have to redirect '/resource/'
requests to the CPVM "public IP" (if you are emulating the public IP range).
We are working on a patch that will allow operators to define an IP address for
this configuration. As soon we open the PR, we notify you.
Best regards,
Daniel Salvador
On 03/10/2022 16:58, Aufgabe Zwei wrote:
Hello,
I have finally been able to setup cloudstack on Ubuntu 20.04, single host.
I am able to ping the system vm's and my created vm instance.
My problem is that I cannot ssh into the vm instance. I see that I should
be able to login using the console proxy url but I am not sure which IP
address the domain should point to.
For what it's worth, I'm currently working on a private cloud setup for
testing before I change to public setup later
Thanks in advance.
Re: Access to disk offerings for different domains
Hi Chris,
Regarding your first doubt:
- while creating disk offerings one can define storage tags to indicate
in which storage (that matches with the tags) the volume should be
created. Also, one can define the disk offering as not public and
dedicate it to a set of domains. This way, the domain will have access
only to a set of offerings that will direct the volumes to the specific
storage. Bare in mind that volumes with disk offerings without storage
tags can be created in any storage, regardless of the storage tags of
the storage.
- if you do not want to use the shipped disk offerings, you can remove them.
- system offerings are not affected by the disk offerings you see via UI
or listing via API. When you create a compute/system offering, they are
automatically shipped with their own disk offering, which cannot be used
individually.
About your second doubt:
- If the offerings are marked as public, they are accessible by anyone
who can list the offerings. The data is stored in the tables
"cloud.disk_offering_details" for disk offerings,
"cloud.service_offering_details" for compute and service offerings, and
"cloud.network_offering_details" for network offerings. If there are
entries in these tables ("cloud.disk_offering_details",
"cloud.service_offering_details", and "cloud.network_offering_details")
with the name "domainid", it means the offering is dedicated to the
domains; otherwise, it is public.
- If you are seeking for permissions to the APIs, they are based on the
role the user is using. The table storing the role permission is
"cloud.role_permissions".
Best regards,
Daniel Salvador
On 05/09/2022 07:32, vas...@gmx.de wrote:
Hi,
Overall Usecase:
- we want to provide separate storage pools for different domains
- we want to provide customized disk offerings for each domain which use
the different storage pools and prevent usage of the "shipped" disk
offerings. These shall stay in place for use with SystemVMs /
SystemOfferings
Has anyone a similar usecase or give "best practices" for such a setup?
Nevertheless I am currently looking into the access rights for service
offerings - especially the "shipped" one wich are availeable out of the box.
Therefore for getting a better understanding i am looking for the tables
where a "change" of access settings is stored.
Could anyone point out the tables where these "access rights" are stored?
Regards,
Chris
Re: Add certificate SSL ConsoleProxy
Hi, Jonathan, It is a known bug, fixed with PR https://github.com/apache/cloudstack/pull/5682 in version 4.16.1.0. However, if you are not using ACS 4.16.1.0 or higher, there is a workaround for the situation: after filling the fields, set focus to the last field (*DNS domain suffix (i.e., xyz.com)*) and press Enter. It will submit the formulary to the backend. Best regards. On 13/06/2022 19:17, Jonathan Farias Valenzuela wrote: Hi, Please help me with the following problem: I'm trying to add the ssl certificates for the proxy console, when adding in the UI I can't advance when I submit. Jonathan Farias V.|***Ingeniero DataCenter*| *SIXMANAGER* Avda. Américo Vespucio Sur 1307, Oficina 404, Las Condes, Santiago - Chile
Re: Dynamic VM Scaling KVM
Hi Abishek,
It's only available via CLI (at least, right now). There are some
details that you need to pay attention when dynamic scaling the VMs
(with KVM) and using the API:
(examples via CloudMonkey)
- The definitions that allow a VM (in KVM) to be dynamic scalable are
set in the XML while deploying the VM. Therefore, if you have a running
VM or change its service offering, you will need to stop it and start it
again.
- If you are using a custom constrained service offering, you always
have to inform the parameters 'details[0].memory' and
'details[0].cpuNumber='; also, if you are using a custom unconstrained
service offering, you must inform the parameter 'details[0].cpuSpeed'.
If you only want to increase the memory, you will need to inform the
current CPU number and vice-versa.
- Related to memory, as they are hotplugged, the value to be added must
be multiple of 128 (Mb). Therefore, if your VM has 1500Mb of memory, for
instance, and you want to add some more, the value informed in the API
must be the current value + some number multiple of 128. For instance,
the parameter 'details[0].memory' must receive the value '1628' (1500 +
128), '1756' (1500 + 256), '1884' (1500 + 384), '2012' (1500 + 512) and
so on.
Best regards,
Daniel
On 21/11/2021 03:49, Abishek wrote:
Hello All,
I have successfully upgraded qemu-kvm to qemu-kvm-ev and now I can
start the VM with dynamic-scaling template and
service-offering(custom) unlike before. But after
successfully starting the VM I did not find any scaleVM option in the
Web UI while the VM is running. Is the feature only supported via CLI
only or do I have to enable some other options (enable dynamic scale
vm is already set to true).
Mgmt : Cloudstack 4.16 (upgraded from 4.15.2)
HV: Centos 7.9 with KVM
Thank You
image.png
On Fri, 19 Nov 2021 at 19:18, Abishek wrote:
Hi Slavka,
I am very grateful for the response. Can i directly proceed to
replace qemu-kvm with qemu-kvm-ev package. What effect will it
have on my current environment? If you could point me to any
documentation regarding it I would be extremely grateful.
Environment:
Hypervisor: Centos 7.9 KVM
Thank You.
On Fri, 19 Nov 2021 at 16:27, Slavka Peleva
wrote:
Hi all,
I've tested the dynamic scaling with qemu-kvm and qemu-kvm-ev.
Here are the
results for both with "Custom constrained" compute offering:
qemu-kvm-ev
deploy VM - passed
scale VM - passed
qemu-kvm
deploy VM - failed with the same error
(org.libvirt.LibvirtException:
internal error: process exited while connecting to monitor:
qemu: invalid
ram size: size=524288k,slots=16,maxmem=4194304k)
scale VM on existing VM - passed
Best regards,
Slavka
On Fri, Nov 19, 2021 at 6:12 AM Abishek
wrote:
> Thanks a lot Wei for helping out. I will first try it in the
Ubuntu
> environment.
>
> On Fri, 19 Nov 2021 at 00:09, Wei ZHOU
wrote:
>
> > Great, thanks Daniel.
> >
> > Abishek, now it seems you need to install qemu-kvm-ev for
testing the
> > feature.
> >
> > -Wei
> >
> > On Thursday, 18 November 2021, Daniel Augusto Veronezi
Salvador <
> > dvsalvador...@gmail.com> wrote:
> >
> > > Hi Abishek, Wei, I tested this feature with Ubuntu 20.04
+ KVM. Some
> > > community members tested with CentOS7 and qemu-kvm-ev
and it worked.
> > Also,
> > > as live snapshots and live migrations, hotplug memory is
a known
> > limitation
> > > of qemu-kvm. I cannot give you advice regarding this
replacement too,
> as
> > I
> > > do not use CentOS. Best regards, Daniel Salvador
> > >
> > > On 17/11/2021 09:07, Wei ZHOU wrote:
> > >
> > >> Hi Abishek,
> > >>
> > >> I cannot give your advice on it as I did not install
qemu-kvm-ev to
> > >> replace
> > >> qemu-kvm before.
> > >>
> > >> I suspect that the issue you faced is caused by
qemu-kvm. qemu-kvm-ev
> > >> supports more advanced features than qemu-kvm.
> > >>
> > >> cc @Daniel, what hypervisor did you use to test the
dynamic scaling ?
> > >>
> > >> -Wei
> > >>
> >
Re: Dynamic VM Scaling KVM
Hi Abishek, Wei, I tested this feature with Ubuntu 20.04 + KVM. Some community members tested with CentOS7 and qemu-kvm-ev and it worked. Also, as live snapshots and live migrations, hotplug memory is a known limitation of qemu-kvm. I cannot give you advice regarding this replacement too, as I do not use CentOS. Best regards, Daniel Salvador On 17/11/2021 09:07, Wei ZHOU wrote: Hi Abishek, I cannot give your advice on it as I did not install qemu-kvm-ev to replace qemu-kvm before. I suspect that the issue you faced is caused by qemu-kvm. qemu-kvm-ev supports more advanced features than qemu-kvm. cc @Daniel, what hypervisor did you use to test the dynamic scaling ? -Wei On Wed, 17 Nov 2021 at 11:42, Abishek wrote: HI Wei, I have not tested with qemu-kvm-ev. I am willing to try but what effect will it have on my existing setup when I install qemu-kvm-ev in my existing qemu-kvm. Should I have to remove qemu-kvm and install qemu-kvm-ev? In the previous test I was able to start the machine with Fixed Offering but the option for Scaling the VM in UI was not present while the VM was in running state but was available when the VM was in shutdown state only. Thank You. On Wed, 17 Nov 2021 at 15:23, Wei ZHOU wrote: Hi Abishek, Have you tested with qemu-kvm-ev instead of qemu-kvm ? -Wei On Wed, 17 Nov 2021 at 10:00, Abishek wrote: Hello Wei, I am very grateful for your response. In my test environment I first created a service offering of Custom Constrained (minCore=1, maxCore=10, minRam=1GB, maxRam=12GB, CPUSpeed=2000) with dynamic scaling enabled. But every time I tried to start the VM under this ServiceOffering I got the above mentioned error. Please find the following agent log for that particular VM: https://controlc.com/4618c4d8 HOST OS : Centos 7.9 QEMU: QEMU emulator version 1.5.3 (qemu-kvm-1.5.3-175.el7_9.4), Copyright (c) 2003-2008 Fabrice Bellard And I think the error seems to be with the service offering (Custom Constrained). So, I created two Fixed services offering A(1 core and 1GB) and B(2 Core and 2 GB) with dynamic scaling enabled. I then built a VM based on service offering A and the VM started(finally). But I did not find any option to scale the VM while it's running. I was only able to scale the VM to shutdown state. Thank You. On Wed, 17 Nov 2021 at 13:11, Wei ZHOU wrote: Hi Abishek, I guess the error is same as before: qemu: invalid ram size: size=4194304k,slots=16,maxmem=1583660032k What are your hypervisor/qemu version and guest OS version ? Can you check agent.log and share the xml definition of the vm ? -Wei On Wed, 17 Nov 2021 at 05:18, Abishek Budhathoki Hello Wei, I have also tested with limited service offering where max RAM is 12GB and CPU is 12 core. But the result is the same. I can not start the VM. But if I turn off Dynamic Scaling to Off on Advance Mode while creating the instance I can start the VM. Thank You. On Tue, 16 Nov 2021 at 20:50, Wei ZHOU wrote: The max ram looks very huge. It would be better to double check the offerings. "minRam":"(4.00 GB) 4294967296","maxRam":"(1.4749 TB) 1621666836480" -Wei On Tue, 16 Nov 2021 at 15:21, Abishek Budhathoki < abckd...@gmail.com wrote: HI Daniel, Extremely glad to hear that dynamic scaling is now supported in KVM. There is no issue of resources. I cannot start a VM with dynamic scaling enabled template and service offering. But I am able to start other VMs. I have shared the logs on https://controlc.com/204f727d Thank You On Tue, 16 Nov 2021 at 16:51, Abishek Budhathoki < abckd...@gmail.com wrote: Hello All, I have just upgraded my Cloudstack installation from 4.15 to 4.16 and everything is working as expected. In the docs site(Changes in 4.16.0.0 since 4.15) I have seen that dynamic Scaling with KVM now works with Cloudstack 4.16(github #4878 Support vm dynamic scaling with kvm). Is this true? I have tried to test this but was unsuccessful. Changed global setting enable dynamic scaling to true, created template with dynamic scaling support, created service offering that supports dynamic Scaling. I only see the following error: ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-12:ctx-bb24d685 job-2304) (logid:394001c9) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to insufficient capacity If it's not still supported please discard this email. Thank You.
Re: Dynamic VM Scaling KVM
Hi Abishek, It's already supported. The error you reported seems to be related to (according to logs presented by you) insufficient capacity of the environment when deploying the VM. It could be related to a lack of network (like DHCP) or compute (like RAM or CPU) resources, could you provide more logs? Also, what happens if you try to deploy a VM with the same resources, but without dynamic scaling? Best regards, Daniel Salvador On 16/11/2021 08:06, Abishek Budhathoki wrote: Hello All, I have just upgraded my Cloudstack installation from 4.15 to 4.16 and everything is working as expected. In the docs site(Changes in 4.16.0.0 since 4.15) I have seen that dynamic Scaling with KVM now works with Cloudstack 4.16(github #4878 Support vm dynamic scaling with kvm). Is this true? I have tried to test this but was unsuccessful. Changed global setting enable dynamic scaling to true, created template with dynamic scaling support, created service offering that supports dynamic Scaling. I only see the following error: ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-12:ctx-bb24d685 job-2304) (logid:394001c9) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to insufficient capacity If it's not still supported please discard this email. Thank You.
Re: All cluster reboot when a Primary storage fails
Hi Mauro, On KVM's monitor, when there is an inconsistency on the heartbeat's file or heartbeat timeout is extrapolated several times, by default, the host is restarted. The PR 4586 (https://github.com/apache/cloudstack/pull/4586) already addressed this issue by externalizing a property, which allows the operator to decide if the host must be restarted or not (default is 'true', meaning that the host will be restarted). However, this feature will be available only after release 4.16. Best regards, Daniel Salvador On 15/10/2021 20:43, Mauro Ferraro - G2K Hosting wrote: Hi guys, how are you?. We are having this problems with ACS when a primary storages fails. We have several primary storage with Linux and NFS server serving KVM images. So every hosts have been mounted all the NFS servers because in one Host can be running VMs from different storages. The main problem of this, is when some storage fails because any reason all the cluster gets crazy and start rebooting the hosts to reconnect with this storage and all the VMs on the cluster, (including the VMs that were working good) goes down becuase the conection to one storage fails. If the problem with storage is permanent, the cluster never start again and hosts will reboot indefinitely. When this problem appears, the logs say this: host heartbeat: kvmheartbeat.sh will reboot system because it was unable to write the heartbeat to the storage. Many users, edit the script kvmheartbeat.shto avoid the hosts reboot or restart the agent on the host but i really not be sure that this is the real solution. Can someone help to propose a best solution at this high risk problem?. Regards, Mauro
Re: The RAM MEM inside KVM instance is not the same as it was set
Hi, Cristian,
When ACS is deploying a VM, it defines a minimum memory|CPU to the VM, based on
a calculation through the overprovisioning (defined mem|cpu speed /
overprovisioning factor); Therefore, it will define a range of memory|CPU, even
if it does not use scalable (dynamic) service offerings.
For now, the option to avoid this behavior is to keep memory overprovisioning
as "1", as Wei suggested. However, in version 4.16.0, there'll be two new
global settings
("vm.min.memory.equals.memory.divided.by.mem.overprovisioning.factor" and
"vm.min.cpu.speed.equals.cpu.speed.divided.by.cpu.overprovisioning.factor") to
avoid it.
This is the PR which introduced it:
https://github.com/apache/cloudstack/pull/4722
Best regards,
Daniel
On 2021/09/17 15:07:55, Wei ZHOU wrote:
> For kvm environments, I suggest to set the value to 1 (in global setting
> and cluster setting).
>
> -Wei
>
>
> On Fri, 17 Sept 2021 at 16:19, wrote:
>
> > Hi,
> >
> > Yes, I have found that this is why i have this value.
> >
> >
> > Regards,
> > Cristian
> >
> > -Original Message-
> > From: Wei ZHOU
> > Sent: Friday, September 17, 2021 3:34 PM
> > To: users
> > Subject: Re: The RAM MEM inside KVM instance is not the same as it was set
> >
> > Hi,
> >
> > Did you set mem.overprovisioning.factor in global setting or cluster
> > setting ?
> >
> > -Wei
> >
> > On Fri, 17 Sept 2021 at 13:47, wrote:
> >
> > > Hello,
> > >
> > >
> > >
> > >From where do I get this minRam value? I used a compute offering
> > > with 10GB of RAM, if I check the VM I see 4GB Ram.
> > >
> > >
> > >
> > >
> > >
> > "[{"com.cloud.agent.api.StartAnswer":{"vm":{"id":"4156","name":"i-2-4156-VM"
> > >
> > > ,"state":"Starting","type":"User","cpus":"4","minSpeed":"760","maxSpee
> > > d":"19
> > > 00","minRam":"(4.00 GB) 4294967296","maxRam":"(10.00 GB)
> > > 10737418240","arch":"x86_64","os":"CentOS
> > > 7","platformEmulator":"CentOS
> > >
> > > 7","bootArgs":"","enableHA":"false","limitCpuUse":"false","enableDynam
> > > ically
> > >
> > > ScaleVm":"false","vncPassword":"NutaRZIRWThdE3aU_IkJAA","vncAddr":"10.
> > > 100.22
> > > .10""
> > >
> > >
> > >
> > >
> > >
> > > Regards,
> > >
> > > Cristian
> > >
> > >
> >
> >
>
Re: [VOTE] Standard string lib
Hi Mario, thanks for the reply. This was sent to the wrong mail list. If you want to join the voting, please, follow the official thread in DEV mail list: https://lists.apache.org/thread.html/rf8c21f7d8af2c85369c7ce2f8205dc8f1b24ebd0a24417ba501b79a0%40%3Cdev.cloudstack.apache.org%3E Best regards, Daniel On 2021/09/13 14:58:04, mario juliano grande-balletta wrote: > +1 > > yes, go for it! > > good idea > > > > > > > > On Mon, 2021-09-13 at 14:54 +, Daniel Augusto Veronezi Salvador > wrote: > > Hi All, > > > > We had a discussion about standardizing the string libs we're using > > (https://lists.apache.org/thread.html/r806cd10b3de645c150e5e0e3d845c5a380a700197143f57f0834d758%40%3Cdev.cloudstack.apache.org%3E). > > > > As I proposed, I'm opening this voting thread to see if all are in favor of > > using "commons.lang3" as the String standard library and for String > > operations not convered on "commons.lang3", we use our StringUtils. Then, > > if the vote passes, I will create the PR to address this change in the code > > base by removing unnecessary libraries, and changing the code to use > > "commons.lang3". > > > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove (and reason why) > > > > Best regards, > > Daniel > > > > >
Re: [VOTE] Standard string lib
Hi all, I sent it to the wrong mail list, please disregard this thread, thanks. Best regards, Daniel On 13/09/2021 11:54, Daniel Augusto Veronezi Salvador wrote: Hi All, We had a discussion about standardizing the string libs we're using (https://lists.apache.org/thread.html/r806cd10b3de645c150e5e0e3d845c5a380a700197143f57f0834d758%40%3Cdev.cloudstack.apache.org%3E). As I proposed, I'm opening this voting thread to see if all are in favor of using "commons.lang3" as the String standard library and for String operations not convered on "commons.lang3", we use our StringUtils. Then, if the vote passes, I will create the PR to address this change in the code base by removing unnecessary libraries, and changing the code to use "commons.lang3". [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and reason why) Best regards, Daniel
[VOTE] Standard string lib
Hi All, We had a discussion about standardizing the string libs we're using (https://lists.apache.org/thread.html/r806cd10b3de645c150e5e0e3d845c5a380a700197143f57f0834d758%40%3Cdev.cloudstack.apache.org%3E). As I proposed, I'm opening this voting thread to see if all are in favor of using "commons.lang3" as the String standard library and for String operations not convered on "commons.lang3", we use our StringUtils. Then, if the vote passes, I will create the PR to address this change in the code base by removing unnecessary libraries, and changing the code to use "commons.lang3". [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and reason why) Best regards, Daniel
Re: NFS Error Ubuntu 20.4
Hello John,
This mount is done by the SSVM. Does the SSVM has access to the NFS you
trying to use as a secondary storage?
Best regards,
Daniel Salvador
On 08/07/2021 08:53, John wrote:
We are having some issues with the latest CS 4.15.1 which we never had
before.
We are using the following:-
Ubuntu 20.4
CS 4.15.1
NFS is on the same host as the KVM server.
We can mount the /export/secondary directory successfully from the
management host.
But when CS tries to mount it we get this error is the management log.
Unable to mount XXX.XXX.:/export/secondary at
/mnt/SecStorage/4891ed6e-23b4-3b63-a190-1298ce00ed8d due to mount.nfs:
access denied by server while mounting XXX.XXX.:/export/secondary
Further, as we are using the KVM host as the NFS server do we need to
install NFS on the management host as well as the KVM host ?
Here is some further logs which may assist.
2021-07-08 11:36:48,252 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-9:null) (logid:) Seq 2-5813865643958599684:
Processing: { Ans: , MgmtId: 148407405838237, via: 2, Ver: v1, Flags:
110,
[{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException:
GetRootDir for nfs://XX.XX.XX.XX/export/secondary
failed due to
com.cloud.utils.exception.CloudRuntimeException: Unable to
mountXX.XX.XX.XX:/export/secondary at
/mnt/SecStorage/4891ed6e-23b4-3b63-a190-1298ce00ed8d due to mount.nfs:
parsing error on 'vers=' option
Any tips would be helpful.
Sent with ProtonMail
Re: Downloading template 403 Forbidden Error
Hello, To access the Secondary Storagem VM, you need to connect in the SSVM's host and use the command *cloudstack-ssh, *with the IP available in *Link Local IP Address:***cloudstack-ssh x.x.x.x Best regards, Guto Veronezi On 05/07/2021 22:22, 조대형 wrote: Hi All. CloudStack 4.15.0.0 I have an error which is 403 Forbidden when I tried to download templates. Secondarystoragevm is up and running. Cloudstack logs : 2021-07-06 10:11:45,657 WARN [c.c.a.AlertManagerImpl] (RouterStatusMonitor- 1:ctx-d493e8c5) (logid:204ee303) AlertType:: 9 | d ataCenterId:: 1 | podId:: 1 | clusterId:: null | message:: Health checks failed: 1 failing checks on router dd14de61-88c1-41f0 - 9b99-28f1ee4d13d2 2021-07-06 10:11:45,672 WARN [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-d493e8c5) (logid:204ee30 3) Health checks failed: 1 failing checks on router dd14de61-88c1-41f0-9b99- 28f1ee4d13d2. Checking failed health checks to see if router needs recreate 2021-07-06 10:13:36,727 WARN [c.c.a.d.ParamGenericValidationWorker] (qtp1026871825-772241:ctx-1b93af0a ctx-5458a69e) (logid:a 9828694) Received unknown parameters for command listZones. Unknown parameters : listall 2021-07-06 10:13:39,167 WARN [c.c.a.ApiServer] (qtp1026871825-778326:ctx- 29a17f02 ctx-3e6dcf5c) (logid:e31434c2) Unknown API command: listAndSwitchSamlAccount Do you have ideas? Thanks,
Re: Downloading template 403 Forbidden Error
Hello, Could you access the Secondary Storage VM and try to retrieve this content (via CURL, WGET...)? Best regards, Guto Veronezi On 05/07/2021 22:22, 조대형 wrote: Hi All. CloudStack 4.15.0.0 I have an error which is 403 Forbidden when I tried to download templates. Secondarystoragevm is up and running. Cloudstack logs : 2021-07-06 10:11:45,657 WARN [c.c.a.AlertManagerImpl] (RouterStatusMonitor- 1:ctx-d493e8c5) (logid:204ee303) AlertType:: 9 | d ataCenterId:: 1 | podId:: 1 | clusterId:: null | message:: Health checks failed: 1 failing checks on router dd14de61-88c1-41f0 - 9b99-28f1ee4d13d2 2021-07-06 10:11:45,672 WARN [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-d493e8c5) (logid:204ee30 3) Health checks failed: 1 failing checks on router dd14de61-88c1-41f0-9b99- 28f1ee4d13d2. Checking failed health checks to see if router needs recreate 2021-07-06 10:13:36,727 WARN [c.c.a.d.ParamGenericValidationWorker] (qtp1026871825-772241:ctx-1b93af0a ctx-5458a69e) (logid:a 9828694) Received unknown parameters for command listZones. Unknown parameters : listall 2021-07-06 10:13:39,167 WARN [c.c.a.ApiServer] (qtp1026871825-778326:ctx- 29a17f02 ctx-3e6dcf5c) (logid:e31434c2) Unknown API command: listAndSwitchSamlAccount Do you have ideas? Thanks,
Re: Snapshots are not working after upgrading to 4.15.0
Hello Andrei, As yourself noticed, ACS has a hardcoded threshold for secondary storages. In cases that the secondary storage has large capacities, 10% can mean a lot of storage. There is an open PR (https://github.com/apache/cloudstack/pull/4790) that externalize this threshold to allow operators decide how much they need. Also, the logs of secondary storage management were improved with PR https://github.com/apache/cloudstack/pull/4955. With respect to KVM snapshots, volume snapshots are taken in a quite peculiar way. Instead of taking volume snapshots directly, ACS takes a full snapshot of the VM, which may cause freeze on it (VM) due to memory snapshot, and then extracts the disk from the VM snapshot. Due to it, it was open an issue https://github.com/apache/cloudstack/issues/5124 to discuss a new workflow to snapshots on KVM. I am already implementing the solution for the issue 5124, and to improve this whole snapshot process for KVM; However, it is a complex and long standing job. As soon as we have something, I would appreciate to receive some feedback from you. Regards, Guto On 2021/06/16 16:15:51, Andrei Mikhailovsky wrote: > Hello,> > > I've done some more investigation and indeed, the snapshots were not taken because the secondary storage was over 90% used. I have started cleaning some of the older volumes and noticed another problem. After removing snapshots, they do not seem to be removed from the secondary storage. I've removed all snapshots over 24 hours ago and it looks like the disk space hasn't been freed up at all.> > > Looks like there are issues with snapshotting function after all.> > > Andrei> > > > > - Original Message -> > > From: "Harikrishna Patnala" > > > To: "users" > > > Sent: Tuesday, 8 June, 2021 03:33:57> > > Subject: Re: Snapshots are not working after upgrading to 4.15.0> > > > Hi Andrei,> > > > > > Can you check the following things and let us know?> > > > > > > > > 1. Can you try creating a new volume and then create snapshot of that, to check> > > if this an issue with old entries> > > 2. For the snapshots which are failing can you check if you are seeing any> > > error messages like this "Can't find an image storage in zone with less than".> > > This is to check if secondary storage free space check failed.> > > 3. For the snapshots which are failing and if it is delta snapshot can you> > > check if its parent's snapshot entry exists in "snapshot_store_ref" table with> > > 'parent_snapshot_id' of the current snapshot with 'store_role' "Image". This is> > > to find the secondary storage where the parent snapshot backup is located.> > > > > > Regards,> > > Harikrishna> > > > > > From: Andrei Mikhailovsky > > > Sent: Monday, June 7, 2021 7:00 PM> > > To: users > > > Subject: Snapshots are not working after upgrading to 4.15.0> > > > > > Hello everyone,> > > > > > I am having an issue with volume snapshots since I've upgraded to 4.15.0. None> > > of the volumes are being snapshotted regardless if the snapshot is initiated> > > manually or from the schedule. The strange thing is that if I manually take the> > > snapshot, the GUI shows Success status, but the Storage>Snapshots show an Error> > > status. Here is what I see in the management server logs:> > > > > > 2021-06-07 13:55:20,022 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]> > > (Work-Job-Executor-81:ctx-08dd4222 job-86141/job-86143) (logid:be34ce01) Done> > > executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-86143> > > 2021-06-07 13:55:20,024 INFO [o.a.c.f.j.i.AsyncJobMonitor]> > > (Work-Job-Executor-81:ctx-08dd4222 job-86141/job-86143) (logid:be34ce01) Remove> > > job-86143 from job monitoring> > > 2021-06-07 13:55:20,094 DEBUG [o.a.c.s.s.SnapshotServiceImpl]> > > (BackupSnapshotTask-3:ctx-744796da) (logid:607dbb0e) Failed to copy snapshot> > > com.cloud.utils.exception.CloudRuntimeException: can not find an image stores> > > at> > > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:271)> > > at> > > org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171)> > > at> > > com.cloud.storage.snapshot.SnapshotManagerImpl$BackupSnapshotTask.runInContext(SnapshotManagerImpl.java:1238)> > > at> > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)> > > at> > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)> > > at> > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)> > > at> > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)> > > at> > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)> > > at> > > java.base/java.util.conc
