--- Begin Message ---
Hi Stefan,
El 14/4/21 a las 13:22, Stefan M. Radman escribió:
Hi Eneko
Do you have separate physical interfaces for the cluster (corosync)
traffic?
No.
Do you have them on separate VLANs on your switches?
Onyl Ceph traffic is on VLAN91, the rest is untagged.
Are you running 1 or 2 corosync rings?
This is standard... no hand tuning:
nodelist {
node {
name: proxmox1
nodeid: 2
quorum_votes: 1
ring0_addr: 192.168.90.11
}
node {
name: proxmox2
nodeid: 1
quorum_votes: 1
ring0_addr: 192.168.90.12
}
node {
name: proxmox3
nodeid: 3
quorum_votes: 1
ring0_addr: 192.168.90.13
}
}
quorum {
provider: corosync_votequorum
}
totem {
cluster_name: CLUSTERNAME
config_version: 3
interface {
linknumber: 0
}
ip_version: ipv4-6
secauth: on
version: 2
}
Please post your /etc/network/interfaces and explain which interface
connects where.
auto lo
iface lo inet loopback
iface ens2f0np0 inet manual
# Switch2
iface ens2f1np1 inet manual
# Switch1
iface eno1 inet manual
iface eno2 inet manual
auto bond0
iface bond0 inet manual
bond-slaves ens2f0np0 ens2f1np1
bond-miimon 100
bond-mode active-backup
bond-primary ens2f0np1
auto bond0.91
iface bond0.91 inet static
address 192.168.91.11
#Ceph
auto vmbr0
iface vmbr0 inet static
address 192.168.90.11
gateway 192.168.90.1
bridge-ports bond0
bridge-stp off
bridge-fd 0
Thanks
Thanks
Stefan
On Apr 14, 2021, at 12:12, Eneko Lacunza via pve-user
<[email protected] <mailto:[email protected]>> wrote:
*From: *Eneko Lacunza <[email protected] <mailto:[email protected]>>
*Subject: **Re: [PVE-User] PVE 6.2 Strange cluster node fence*
*Date: *April 14, 2021 at 12:12:09 GMT+2
*To: *[email protected] <mailto:[email protected]>
Hi Michael,
El 14/4/21 a las 11:21, Michael Rasmussen via pve-user escribió:
On Wed, 14 Apr 2021 11:04:10 +0200
Eneko Lacunza via pve-user<[email protected]
<mailto:[email protected]>> wrote:
Hi all,
Yesterday we had a strange fence happen in a PVE 6.2 cluster.
Cluster has 3 nodes (proxmox1, proxmox2, proxmox3) and has been
operating normally for a year. Last update was on January 21st 2021.
Storage is Ceph and nodes are connected to the same network switch
with active-pasive bonds.
proxmox1 was fenced and automatically rebooted, then everything
recovered. HA restarted VMs in other nodes too.
proxmox1 syslog: (no network link issues reported at device level)
I have seen this occasionally and every time the cause was high network
load/network congestion which caused token timeout. The default token
timeout in corosync IMHO is very optimistically configured to 1000 ms
so I have changed this setting to 5000 ms and after I have done this I
have never seen fencing happening caused by network load/network
congestion again. You could try this and see if that helps you.
PS. my cluster communication is on a dedicated gb bonded vlan.
Thanks for the info. In this case network is 10Gbit (I see I didn't
include this info) but only for proxmox nodes:
- We have 2 Dell N1124T 24x1Gbit 4xSFP+ switches
- Both switches are interconnected with a SFP+ DAC
- Active-passive Bonds in each proxmox node go one SFP+ interface on
each switch. Primary interfaces are configured to be on the same switch.
- Connectivity to the LAN is done with 1 Gbit link
- Proxmox 2x10G Bond is used for VM networking and Ceph
public/private networks.
I wouldn't expect high network load/congestion because it's on an
internal LAN, with 1Gbit clients. No Ceph issues/backfilling were
ocurring during the fence.
Network cards are Broadcom.
Thanks
Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es <https://www.binovo.es>
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO
<https://www.youtube.com/user/CANALBINOVO>
https://www.linkedin.com/company/37269706/
_______________________________________________
pve-user mailing list
[email protected]
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-user&data=04%7C01%7Csmr%40kmi.com%7C94935b3774c84a829c8008d8ff2dcd78%7Cc2283768b8d34e008f3d85b1b4f03b33%7C0%7C0%7C637539919485970079%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=0Lc31YKv%2Fm4RQEsAZlcdsuA1XidEZEgfmAwRgGT4Dlg%3D&reserved=0
CONFIDENTIALITY NOTICE: /This communication may contain privileged and
confidential information, or may otherwise be protected from
disclosure, and is intended solely for use of the intended
recipient(s). If you are not the intended recipient of this
communication, please notify the sender that you have received this
communication in error and delete and destroy all copies in your
possession. /
Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/
--- End Message ---
_______________________________________________
pve-user mailing list
[email protected]
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
