Thanks to all guys for your feedback.
We think that the problem is hard to solve without damage a VM. We were
trying with Gluster+NFS Ganesha+PaceMaker+Corosync and when the NFS goes
down de IP floats to other node but it takes few seconds, and all VMs
goes down and in this scenario the VM be Damaged. And the performance
with gluster is not really good.
Now we want to test with ACS 4.16 and linstor, have someone any
references about this?.
Regards
Mauro
El 20/10/2021 a las 05:44, Piotr Pisz escribió:
Hi,
I experienced this problem myself, in the KVM, Ceph, NFS-Ganesha environment at
full Ceph load, the Ganesha NFS server was able to hang.
Servers were able to randomly restart due to lack of NFS access.
Which magnified the problem and there was a cascade and restart of the entire
environment.
We currently have the restart line removed in kvmheartbeat, instead we report
the restart attempt via prometheus.
Regards,
Piotr
-----Original Message-----
From: Sina Kashipazha <s.kashipa...@protonmail.com.INVALID>
Sent: Wednesday, October 20, 2021 10:35 AM
To: users@cloudstack.apache.org
Subject: Re: All cluster reboot when a Primary storage fails
Hey Daniel,
PR #4586 (https://github.com/apache/cloudstack/pull/4586) addressed your issue,
as well. I'm currently working on it. Could you share with me how I can
reproduce your reboot problem?
Kind regards,
Sina
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Saturday, October 16th, 2021 at 05:40, Daniel Augusto Veronezi Salvador
<dvsalvador...@gmail.com> wrote:
Hi Mauro,
On KVM's monitor, when there is an inconsistency on the heartbeat's file
or heartbeat timeout is extrapolated several times, by default, the host
is restarted.
The PR 4586 (https://github.com/apache/cloudstack/pull/4586) already
addressed this issue by externalizing a property, which allows the
operator to decide if the host must be restarted or not (default is
'true', meaning that the host will be restarted). However, this feature
will be available only after release 4.16.
Best regards,
Daniel Salvador
On 15/10/2021 20:43, Mauro Ferraro - G2K Hosting wrote:
Hi guys, how are you?.
We are having this problems with ACS when a primary storages fails.
We have several primary storage with Linux and NFS server serving KVM
images. So every hosts have been mounted all the NFS servers because
in one Host can be running VMs from different storages. The main
problem of this, is when some storage fails because any reason all the
cluster gets crazy and start rebooting the hosts to reconnect with
this storage and all the VMs on the cluster, (including the VMs that
were working good) goes down becuase the conection to one storage fails.
If the problem with storage is permanent, the cluster never start
again and hosts will reboot indefinitely.
When this problem appears, the logs say this:
host heartbeat: kvmheartbeat.sh will reboot system because it was
unable to write the heartbeat to the storage.
Many users, edit the script kvmheartbeat.shto avoid the hosts reboot
or restart the agent on the host but i really not be sure that this is
the real solution.
Can someone help to propose a best solution at this high risk problem?.
Regards,
Mauro
