Hello,
I run a 5 node PVE cluster with pve-manager/6.4-8/185e14db (running
kernel: 5.4.119-1-pve). The storage backend is a HDD based "external"
ceph cluster running Ceph 14.2.16 with 144 OSDs on 9 hosts. Currently
there are about 70 VMs running on this PVE cluster, all Linux (Ubuntu,
SLES).
The problem I have is that writing on VMS has become slower and slower
over time and eg running linux updates (eg apt upgrade) on the VMS takes
longer and longer. The reason seams to be a steadily rising write IOPs
rate on the storage side. Of course over time the number of VNMs also
increased up to the current number causing higher numbers.
Over the week day I can see rates on the ceph side of up to 1000
IOPS/sec writing and about 300 IOPS/sec reading. The really stange thing
is however that even at weekends where the services the VMs offer are
hardly used at all, there is still a quite high write IOPS rate of about
400/sec whereas the read rate is only about 50 IOPS/sec then. The Bytes
read/written are minimal at this time with only about 100KBytes read/sec
and about 5MBytes write/sec.
So what I am looking for is by what the "always there" write IOPS-Rate
of about 400 could be caused. My guess is that this could be caused by
file time (mtime,ctime,atime) write updates to the VMs filesystems. If
this was true then using lazytime in /etc/fstab on all VMs could help to
avoid this behaviour.
But on the other hand all VMs use the (safe) "Writeback"-cache setting.
So shouldn't this cache mode also cache writes caused by updates for
file times?
If yes, than I have to look for other reasons for my write IOPS problem
allthough I have no idea about this at the moment. Any suggestions?
Thanks
Rainer
--
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287
1001312
_______________________________________________
pve-user mailing list
[email protected]
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
