Some kernels do not like values below 5%, thus I prefer to use vm.dirty_bytes & vm.dirty_background_bytes. Try the following ones (comment out the vdsm.conf values ):vm.dirty_background_bytes = 200000000vm.dirty_bytes = 450000000 It's more like shooting in the dark , but it might help. Best Regards,Strahil Nikolov В неделя, 14 април 2019 г., 19:06:07 ч. Гринуич+3, Alex McWhirter <a...@triadic.us> написа: On 2019-04-13 03:15, Strahil wrote: > Hi, > > What is your dirty cache settings on the gluster servers ? > > Best Regards, > Strahil NikolovOn Apr 13, 2019 00:44, Alex McWhirter <a...@triadic.us> > wrote: >> >> I have 8 machines acting as gluster servers. They each have 12 drives >> raid 50'd together (3 sets of 4 drives raid 5'd then 0'd together as >> one). >> >> They connect to the compute hosts and to each other over lacp'd 10GB >> connections split across two cisco nexus switched with VPC. >> >> Gluster has the following set. >> >> performance.write-behind-window-size: 4MB >> performance.flush-behind: on >> performance.stat-prefetch: on >> server.event-threads: 4 >> client.event-threads: 8 >> performance.io-thread-count: 32 >> network.ping-timeout: 30 >> cluster.granular-entry-heal: enable >> performance.strict-o-direct: on >> storage.owner-gid: 36 >> storage.owner-uid: 36 >> features.shard: on >> cluster.shd-wait-qlength: 10000 >> cluster.shd-max-threads: 8 >> cluster.locking-scheme: granular >> cluster.data-self-heal-algorithm: full >> cluster.server-quorum-type: server >> cluster.quorum-type: auto >> cluster.eager-lock: enable >> network.remote-dio: off >> performance.low-prio-threads: 32 >> performance.io-cache: off >> performance.read-ahead: off >> performance.quick-read: off >> auth.allow: * >> user.cifs: off >> transport.address-family: inet >> nfs.disable: off >> performance.client-io-threads: on >> >> >> I have the following sysctl values on gluster client and servers, >> using >> libgfapi, MTU 9K >> >> net.core.rmem_max = 134217728 >> net.core.wmem_max = 134217728 >> net.ipv4.tcp_rmem = 4096 87380 134217728 >> net.ipv4.tcp_wmem = 4096 65536 134217728 >> net.core.netdev_max_backlog = 300000 >> net.ipv4.tcp_moderate_rcvbuf =1 >> net.ipv4.tcp_no_metrics_save = 1 >> net.ipv4.tcp_congestion_control=htcp >> >> reads with this setup are perfect, benchmarked in VM to be about >> 770MB/s >> sequential with disk access times of < 1ms. Writes on the other hand >> are >> all over the place. They peak around 320MB/s sequential write, which >> is >> what i expect but it seems as if there is some blocking going on. >> >> During the write test i will hit 320MB/s briefly, then 0MB/s as disk >> access time shoot to over 3000ms, then back to 320MB/s. It averages >> out >> to about 110MB/s afterwards. >> >> Gluster version is 3.12.15 ovirt is 4.2.7.5 >> >> Any ideas on what i could tune to eliminate or minimize that blocking? >> _______________________________________________ >> Users mailing list -- users@ovirt.org >> To unsubscribe send an email to users-le...@ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z7F72BKYKAGICERZETSA4KCLQYR3AORR/ > _______________________________________________ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-le...@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/FMB6NCNJL2WKEDWPAM4OJIRF2GIDJUUE/
Just the vdsm defaults vm.dirty_ratio = 5 vm.dirty_background_ratio = 2 these boxes only have 8gb of ram as well, so those percentages should be super small.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/5U6QGARQSLFXMPP2EB57DSEACZ3H5SBY/