[ovirt-users] Re: Tuning Gluster Writes
Hi, What is your dirty cache settings on the gluster servers ? Best Regards, Strahil NikolovOn Apr 13, 2019 00:44, Alex McWhirter wrote: > > I have 8 machines acting as gluster servers. They each have 12 drives > raid 50'd together (3 sets of 4 drives raid 5'd then 0'd together as > one). > > They connect to the compute hosts and to each other over lacp'd 10GB > connections split across two cisco nexus switched with VPC. > > Gluster has the following set. > > performance.write-behind-window-size: 4MB > performance.flush-behind: on > performance.stat-prefetch: on > server.event-threads: 4 > client.event-threads: 8 > performance.io-thread-count: 32 > network.ping-timeout: 30 > cluster.granular-entry-heal: enable > performance.strict-o-direct: on > storage.owner-gid: 36 > storage.owner-uid: 36 > features.shard: on > cluster.shd-wait-qlength: 1 > cluster.shd-max-threads: 8 > cluster.locking-scheme: granular > cluster.data-self-heal-algorithm: full > cluster.server-quorum-type: server > cluster.quorum-type: auto > cluster.eager-lock: enable > network.remote-dio: off > performance.low-prio-threads: 32 > performance.io-cache: off > performance.read-ahead: off > performance.quick-read: off > auth.allow: * > user.cifs: off > transport.address-family: inet > nfs.disable: off > performance.client-io-threads: on > > > I have the following sysctl values on gluster client and servers, using > libgfapi, MTU 9K > > net.core.rmem_max = 134217728 > net.core.wmem_max = 134217728 > net.ipv4.tcp_rmem = 4096 87380 134217728 > net.ipv4.tcp_wmem = 4096 65536 134217728 > net.core.netdev_max_backlog = 30 > net.ipv4.tcp_moderate_rcvbuf =1 > net.ipv4.tcp_no_metrics_save = 1 > net.ipv4.tcp_congestion_control=htcp > > reads with this setup are perfect, benchmarked in VM to be about 770MB/s > sequential with disk access times of < 1ms. Writes on the other hand are > all over the place. They peak around 320MB/s sequential write, which is > what i expect but it seems as if there is some blocking going on. > > During the write test i will hit 320MB/s briefly, then 0MB/s as disk > access time shoot to over 3000ms, then back to 320MB/s. It averages out > to about 110MB/s afterwards. > > Gluster version is 3.12.15 ovirt is 4.2.7.5 > > Any ideas on what i could tune to eliminate or minimize that blocking? > ___ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-le...@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z7F72BKYKAGICERZETSA4KCLQYR3AORR/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FMB6NCNJL2WKEDWPAM4OJIRF2GIDJUUE/
[ovirt-users] Re: Tuning Gluster Writes
On 2019-04-13 03:15, Strahil wrote: Hi, What is your dirty cache settings on the gluster servers ? Best Regards, Strahil NikolovOn Apr 13, 2019 00:44, Alex McWhirter wrote: I have 8 machines acting as gluster servers. They each have 12 drives raid 50'd together (3 sets of 4 drives raid 5'd then 0'd together as one). They connect to the compute hosts and to each other over lacp'd 10GB connections split across two cisco nexus switched with VPC. Gluster has the following set. performance.write-behind-window-size: 4MB performance.flush-behind: on performance.stat-prefetch: on server.event-threads: 4 client.event-threads: 8 performance.io-thread-count: 32 network.ping-timeout: 30 cluster.granular-entry-heal: enable performance.strict-o-direct: on storage.owner-gid: 36 storage.owner-uid: 36 features.shard: on cluster.shd-wait-qlength: 1 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off auth.allow: * user.cifs: off transport.address-family: inet nfs.disable: off performance.client-io-threads: on I have the following sysctl values on gluster client and servers, using libgfapi, MTU 9K net.core.rmem_max = 134217728 net.core.wmem_max = 134217728 net.ipv4.tcp_rmem = 4096 87380 134217728 net.ipv4.tcp_wmem = 4096 65536 134217728 net.core.netdev_max_backlog = 30 net.ipv4.tcp_moderate_rcvbuf =1 net.ipv4.tcp_no_metrics_save = 1 net.ipv4.tcp_congestion_control=htcp reads with this setup are perfect, benchmarked in VM to be about 770MB/s sequential with disk access times of < 1ms. Writes on the other hand are all over the place. They peak around 320MB/s sequential write, which is what i expect but it seems as if there is some blocking going on. During the write test i will hit 320MB/s briefly, then 0MB/s as disk access time shoot to over 3000ms, then back to 320MB/s. It averages out to about 110MB/s afterwards. Gluster version is 3.12.15 ovirt is 4.2.7.5 Any ideas on what i could tune to eliminate or minimize that blocking? ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z7F72BKYKAGICERZETSA4KCLQYR3AORR/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FMB6NCNJL2WKEDWPAM4OJIRF2GIDJUUE/ Just the vdsm defaults vm.dirty_ratio = 5 vm.dirty_background_ratio = 2 these boxes only have 8gb of ram as well, so those percentages should be super small. ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/H4XWDEHYKD2MQUR45QLMMSK6FBX44KIG/
[ovirt-users] Re: Tuning Gluster Writes
On 2019-04-14 12:07, Alex McWhirter wrote: On 2019-04-13 03:15, Strahil wrote: Hi, What is your dirty cache settings on the gluster servers ? Best Regards, Strahil NikolovOn Apr 13, 2019 00:44, Alex McWhirter wrote: I have 8 machines acting as gluster servers. They each have 12 drives raid 50'd together (3 sets of 4 drives raid 5'd then 0'd together as one). They connect to the compute hosts and to each other over lacp'd 10GB connections split across two cisco nexus switched with VPC. Gluster has the following set. performance.write-behind-window-size: 4MB performance.flush-behind: on performance.stat-prefetch: on server.event-threads: 4 client.event-threads: 8 performance.io-thread-count: 32 network.ping-timeout: 30 cluster.granular-entry-heal: enable performance.strict-o-direct: on storage.owner-gid: 36 storage.owner-uid: 36 features.shard: on cluster.shd-wait-qlength: 1 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off auth.allow: * user.cifs: off transport.address-family: inet nfs.disable: off performance.client-io-threads: on I have the following sysctl values on gluster client and servers, using libgfapi, MTU 9K net.core.rmem_max = 134217728 net.core.wmem_max = 134217728 net.ipv4.tcp_rmem = 4096 87380 134217728 net.ipv4.tcp_wmem = 4096 65536 134217728 net.core.netdev_max_backlog = 30 net.ipv4.tcp_moderate_rcvbuf =1 net.ipv4.tcp_no_metrics_save = 1 net.ipv4.tcp_congestion_control=htcp reads with this setup are perfect, benchmarked in VM to be about 770MB/s sequential with disk access times of < 1ms. Writes on the other hand are all over the place. They peak around 320MB/s sequential write, which is what i expect but it seems as if there is some blocking going on. During the write test i will hit 320MB/s briefly, then 0MB/s as disk access time shoot to over 3000ms, then back to 320MB/s. It averages out to about 110MB/s afterwards. Gluster version is 3.12.15 ovirt is 4.2.7.5 Any ideas on what i could tune to eliminate or minimize that blocking? ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z7F72BKYKAGICERZETSA4KCLQYR3AORR/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FMB6NCNJL2WKEDWPAM4OJIRF2GIDJUUE/ Just the vdsm defaults vm.dirty_ratio = 5 vm.dirty_background_ratio = 2 these boxes only have 8gb of ram as well, so those percentages should be super small. ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/H4XWDEHYKD2MQUR45QLMMSK6FBX44KIG/ doing a gluster profile my bricks give me some odd numbers. %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop - --- --- --- 0.00 131.00 us 131.00 us 131.00 us 1 FSTAT 0.01 104.50 us 77.00 us 118.00 us 14 STATFS 0.01 95.38 us 45.00 us 130.00 us 16 STAT 0.10 252.39 us 124.00 us 329.00 us 61 LOOKUP 0.22 55.68 us 16.00 us 180.00 us635 FINODELK 0.43 543.41 us 50.00 us1760.00 us125 FSYNC 1.52 573.75 us 76.00 us5463.00 us422 FXATTROP 97.727443.50 us 184.00 us 34917.00 us 2092 WRITE %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop - --- --- --- 0.00 0.00 us 0.00 us 0.00 us 70 FORGET 0.00 0.00 us 0.00 us 0.00 us 1792 RELEASE 0.00 0.00 us 0.00 us 0.00 us 23422 RELEASEDIR 0.01 126.20 us 80.00 us 210.00 us 20
[ovirt-users] Re: Tuning Gluster Writes
On 2019-04-14 13:05, Alex McWhirter wrote: On 2019-04-14 12:07, Alex McWhirter wrote: On 2019-04-13 03:15, Strahil wrote: Hi, What is your dirty cache settings on the gluster servers ? Best Regards, Strahil NikolovOn Apr 13, 2019 00:44, Alex McWhirter wrote: I have 8 machines acting as gluster servers. They each have 12 drives raid 50'd together (3 sets of 4 drives raid 5'd then 0'd together as one). They connect to the compute hosts and to each other over lacp'd 10GB connections split across two cisco nexus switched with VPC. Gluster has the following set. performance.write-behind-window-size: 4MB performance.flush-behind: on performance.stat-prefetch: on server.event-threads: 4 client.event-threads: 8 performance.io-thread-count: 32 network.ping-timeout: 30 cluster.granular-entry-heal: enable performance.strict-o-direct: on storage.owner-gid: 36 storage.owner-uid: 36 features.shard: on cluster.shd-wait-qlength: 1 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off auth.allow: * user.cifs: off transport.address-family: inet nfs.disable: off performance.client-io-threads: on I have the following sysctl values on gluster client and servers, using libgfapi, MTU 9K net.core.rmem_max = 134217728 net.core.wmem_max = 134217728 net.ipv4.tcp_rmem = 4096 87380 134217728 net.ipv4.tcp_wmem = 4096 65536 134217728 net.core.netdev_max_backlog = 30 net.ipv4.tcp_moderate_rcvbuf =1 net.ipv4.tcp_no_metrics_save = 1 net.ipv4.tcp_congestion_control=htcp reads with this setup are perfect, benchmarked in VM to be about 770MB/s sequential with disk access times of < 1ms. Writes on the other hand are all over the place. They peak around 320MB/s sequential write, which is what i expect but it seems as if there is some blocking going on. During the write test i will hit 320MB/s briefly, then 0MB/s as disk access time shoot to over 3000ms, then back to 320MB/s. It averages out to about 110MB/s afterwards. Gluster version is 3.12.15 ovirt is 4.2.7.5 Any ideas on what i could tune to eliminate or minimize that blocking? ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z7F72BKYKAGICERZETSA4KCLQYR3AORR/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FMB6NCNJL2WKEDWPAM4OJIRF2GIDJUUE/ Just the vdsm defaults vm.dirty_ratio = 5 vm.dirty_background_ratio = 2 these boxes only have 8gb of ram as well, so those percentages should be super small. ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/H4XWDEHYKD2MQUR45QLMMSK6FBX44KIG/ doing a gluster profile my bricks give me some odd numbers. %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop - --- --- --- 0.00 131.00 us 131.00 us 131.00 us 1 FSTAT 0.01 104.50 us 77.00 us 118.00 us 14 STATFS 0.01 95.38 us 45.00 us 130.00 us 16 STAT 0.10 252.39 us 124.00 us 329.00 us 61 LOOKUP 0.22 55.68 us 16.00 us 180.00 us635 FINODELK 0.43 543.41 us 50.00 us1760.00 us125 FSYNC 1.52 573.75 us 76.00 us5463.00 us422 FXATTROP 97.727443.50 us 184.00 us 34917.00 us 2092 WRITE %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop - --- --- --- 0.00 0.00 us 0.00 us 0.00 us 70 FORGET 0.00 0.00 us 0.00 us 0.00 us 1792 RELEASE 0.00 0.00 us 0.00 us 0.00 us 23422 RELEASEDIR 0.01 126.20 u
[ovirt-users] Re: Tuning Gluster Writes
Some kernels do not like values below 5%, thus I prefer to use vm.dirty_bytes & vm.dirty_background_bytes. Try the following ones (comment out the vdsm.conf values ):vm.dirty_background_bytes = 2vm.dirty_bytes = 45000 It's more like shooting in the dark , but it might help. Best Regards,Strahil Nikolov В неделя, 14 април 2019 г., 19:06:07 ч. Гринуич+3, Alex McWhirter написа: On 2019-04-13 03:15, Strahil wrote: > Hi, > > What is your dirty cache settings on the gluster servers ? > > Best Regards, > Strahil NikolovOn Apr 13, 2019 00:44, Alex McWhirter > wrote: >> >> I have 8 machines acting as gluster servers. They each have 12 drives >> raid 50'd together (3 sets of 4 drives raid 5'd then 0'd together as >> one). >> >> They connect to the compute hosts and to each other over lacp'd 10GB >> connections split across two cisco nexus switched with VPC. >> >> Gluster has the following set. >> >> performance.write-behind-window-size: 4MB >> performance.flush-behind: on >> performance.stat-prefetch: on >> server.event-threads: 4 >> client.event-threads: 8 >> performance.io-thread-count: 32 >> network.ping-timeout: 30 >> cluster.granular-entry-heal: enable >> performance.strict-o-direct: on >> storage.owner-gid: 36 >> storage.owner-uid: 36 >> features.shard: on >> cluster.shd-wait-qlength: 1 >> cluster.shd-max-threads: 8 >> cluster.locking-scheme: granular >> cluster.data-self-heal-algorithm: full >> cluster.server-quorum-type: server >> cluster.quorum-type: auto >> cluster.eager-lock: enable >> network.remote-dio: off >> performance.low-prio-threads: 32 >> performance.io-cache: off >> performance.read-ahead: off >> performance.quick-read: off >> auth.allow: * >> user.cifs: off >> transport.address-family: inet >> nfs.disable: off >> performance.client-io-threads: on >> >> >> I have the following sysctl values on gluster client and servers, >> using >> libgfapi, MTU 9K >> >> net.core.rmem_max = 134217728 >> net.core.wmem_max = 134217728 >> net.ipv4.tcp_rmem = 4096 87380 134217728 >> net.ipv4.tcp_wmem = 4096 65536 134217728 >> net.core.netdev_max_backlog = 30 >> net.ipv4.tcp_moderate_rcvbuf =1 >> net.ipv4.tcp_no_metrics_save = 1 >> net.ipv4.tcp_congestion_control=htcp >> >> reads with this setup are perfect, benchmarked in VM to be about >> 770MB/s >> sequential with disk access times of < 1ms. Writes on the other hand >> are >> all over the place. They peak around 320MB/s sequential write, which >> is >> what i expect but it seems as if there is some blocking going on. >> >> During the write test i will hit 320MB/s briefly, then 0MB/s as disk >> access time shoot to over 3000ms, then back to 320MB/s. It averages >> out >> to about 110MB/s afterwards. >> >> Gluster version is 3.12.15 ovirt is 4.2.7.5 >> >> Any ideas on what i could tune to eliminate or minimize that blocking? >> ___ >> Users mailing list -- users@ovirt.org >> To unsubscribe send an email to users-le...@ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z7F72BKYKAGICERZETSA4KCLQYR3AORR/ > ___ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-le...@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/FMB6NCNJL2WKEDWPAM4OJIRF2GIDJUUE/ Just the vdsm defaults vm.dirty_ratio = 5 vm.dirty_background_ratio = 2 these boxes only have 8gb of ram as well, so those percentages should be super small. ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/5U6QGARQSLFXMPP2EB57DSEACZ3H5SBY/
[ovirt-users] Re: Tuning Gluster Writes
On 2019-04-14 17:07, Strahil Nikolov wrote: > Some kernels do not like values below 5%, thus I prefer to use > vm.dirty_bytes & vm.dirty_background_bytes. > Try the following ones (comment out the vdsm.conf values ): > > vm.dirty_background_bytes = 2 > vm.dirty_bytes = 45000 > It's more like shooting in the dark , but it might help. > > Best Regards, > Strahil Nikolov > > В неделя, 14 април 2019 г., 19:06:07 ч. Гринуич+3, Alex McWhirter > написа: > > On 2019-04-13 03:15, Strahil wrote: >> Hi, >> >> What is your dirty cache settings on the gluster servers ? >> >> Best Regards, >> Strahil NikolovOn Apr 13, 2019 00:44, Alex McWhirter >> wrote: >>> >>> I have 8 machines acting as gluster servers. They each have 12 drives >>> raid 50'd together (3 sets of 4 drives raid 5'd then 0'd together as >>> one). >>> >>> They connect to the compute hosts and to each other over lacp'd 10GB >>> connections split across two cisco nexus switched with VPC. >>> >>> Gluster has the following set. >>> >>> performance.write-behind-window-size: 4MB >>> performance.flush-behind: on >>> performance.stat-prefetch: on >>> server.event-threads: 4 >>> client.event-threads: 8 >>> performance.io-thread-count: 32 >>> network.ping-timeout: 30 >>> cluster.granular-entry-heal: enable >>> performance.strict-o-direct: on >>> storage.owner-gid: 36 >>> storage.owner-uid: 36 >>> features.shard: on >>> cluster.shd-wait-qlength: 1 >>> cluster.shd-max-threads: 8 >>> cluster.locking-scheme: granular >>> cluster.data-self-heal-algorithm: full >>> cluster.server-quorum-type: server >>> cluster.quorum-type: auto >>> cluster.eager-lock: enable >>> network.remote-dio: off >>> performance.low-prio-threads: 32 >>> performance.io-cache: off >>> performance.read-ahead: off >>> performance.quick-read: off >>> auth.allow: * >>> user.cifs: off >>> transport.address-family: inet >>> nfs.disable: off >>> performance.client-io-threads: on >>> >>> >>> I have the following sysctl values on gluster client and servers, >>> using >>> libgfapi, MTU 9K >>> >>> net.core.rmem_max = 134217728 >>> net.core.wmem_max = 134217728 >>> net.ipv4.tcp_rmem = 4096 87380 134217728 >>> net.ipv4.tcp_wmem = 4096 65536 134217728 >>> net.core.netdev_max_backlog = 30 >>> net.ipv4.tcp_moderate_rcvbuf =1 >>> net.ipv4.tcp_no_metrics_save = 1 >>> net.ipv4.tcp_congestion_control=htcp >>> >>> reads with this setup are perfect, benchmarked in VM to be about >>> 770MB/s >>> sequential with disk access times of < 1ms. Writes on the other hand >>> are >>> all over the place. They peak around 320MB/s sequential write, which >>> is >>> what i expect but it seems as if there is some blocking going on. >>> >>> During the write test i will hit 320MB/s briefly, then 0MB/s as disk >>> access time shoot to over 3000ms, then back to 320MB/s. It averages >>> out >>> to about 110MB/s afterwards. >>> >>> Gluster version is 3.12.15 ovirt is 4.2.7.5 >>> >>> Any ideas on what i could tune to eliminate or minimize that blocking? >>> ___ >>> Users mailing list -- users@ovirt.org >>> To unsubscribe send an email to users-le...@ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z7F72BKYKAGICERZETSA4KCLQYR3AORR/ >>> > >> ___ >> Users mailing list -- users@ovirt.org >> To unsubscribe send an email to users-le...@ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/FMB6NCNJL2WKEDWPAM4OJIRF2GIDJUUE/ > > Just the vdsm defaults > > vm.dirty_ratio = 5 > vm.dirty_background_ratio = 2 > > these boxes only have 8gb of ram as well, so those percentages should be > super small. > > ___ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-le...@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/5U6QGARQSLFXMPP2EB57DSEACZ3H5SBY/ i will try this, I went in and disabled TCP offload on all the nics, huge performance boost. went from 110MB/s to 240MB/s seq writes, reads lost a bit of performance going down to 680MB/s, but that's a decent trade off. Latency is still really high though, need to work on that. I think some more TCP tuning might help.___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/priva
[ovirt-users] Re: Tuning Gluster Writes
On 2019-04-14 22:47, Alex McWhirter wrote: > On 2019-04-14 17:07, Strahil Nikolov wrote: > >> Some kernels do not like values below 5%, thus I prefer to use >> vm.dirty_bytes & vm.dirty_background_bytes. >> Try the following ones (comment out the vdsm.conf values ): >> >> vm.dirty_background_bytes = 2 >> vm.dirty_bytes = 45000 >> It's more like shooting in the dark , but it might help. >> >> Best Regards, >> Strahil Nikolov >> >> В неделя, 14 април 2019 г., 19:06:07 ч. Гринуич+3, Alex McWhirter >> написа: >> >> On 2019-04-13 03:15, Strahil wrote: >>> Hi, >>> >>> What is your dirty cache settings on the gluster servers ? >>> >>> Best Regards, >>> Strahil NikolovOn Apr 13, 2019 00:44, Alex McWhirter >>> wrote: I have 8 machines acting as gluster servers. They each have 12 drives raid 50'd together (3 sets of 4 drives raid 5'd then 0'd together as one). They connect to the compute hosts and to each other over lacp'd 10GB connections split across two cisco nexus switched with VPC. Gluster has the following set. performance.write-behind-window-size: 4MB performance.flush-behind: on performance.stat-prefetch: on server.event-threads: 4 client.event-threads: 8 performance.io-thread-count: 32 network.ping-timeout: 30 cluster.granular-entry-heal: enable performance.strict-o-direct: on storage.owner-gid: 36 storage.owner-uid: 36 features.shard: on cluster.shd-wait-qlength: 1 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off auth.allow: * user.cifs: off transport.address-family: inet nfs.disable: off performance.client-io-threads: on I have the following sysctl values on gluster client and servers, using libgfapi, MTU 9K net.core.rmem_max = 134217728 net.core.wmem_max = 134217728 net.ipv4.tcp_rmem = 4096 87380 134217728 net.ipv4.tcp_wmem = 4096 65536 134217728 net.core.netdev_max_backlog = 30 net.ipv4.tcp_moderate_rcvbuf =1 net.ipv4.tcp_no_metrics_save = 1 net.ipv4.tcp_congestion_control=htcp reads with this setup are perfect, benchmarked in VM to be about 770MB/s sequential with disk access times of < 1ms. Writes on the other hand are all over the place. They peak around 320MB/s sequential write, which is what i expect but it seems as if there is some blocking going on. During the write test i will hit 320MB/s briefly, then 0MB/s as disk access time shoot to over 3000ms, then back to 320MB/s. It averages out to about 110MB/s afterwards. Gluster version is 3.12.15 ovirt is 4.2.7.5 Any ideas on what i could tune to eliminate or minimize that blocking? ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z7F72BKYKAGICERZETSA4KCLQYR3AORR/ >> >>> ___ >>> Users mailing list -- users@ovirt.org >>> To unsubscribe send an email to users-le...@ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/FMB6NCNJL2WKEDWPAM4OJIRF2GIDJUUE/ >> >> Just the vdsm defaults >> >> vm.dirty_ratio = 5 >> vm.dirty_background_ratio = 2 >> >> these boxes only have 8gb of ram as well, so those percentages should be >> super small. >> >> ___ >> Users mailing list -- users@ovirt.org >> To unsubscribe send an email to users-le...@ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/5U6QGARQSLFXMPP2EB57DSEACZ3H5SBY/ > > i will try this, > > I went in and disabled TCP offload on all the nics, huge performance boost. > went from 110MB/s to 240MB/s seq writes, reads lost a bit of performance > going down to 680MB/s, but that's a decent trade off. Latency is still really > high though, need to work on that. I think some more TCP tuning might help
[ovirt-users] Re: Tuning Gluster Writes
Interesting. Who’s 10g cards and which offload settings did you disable? Did you do that on the servers or the vm host clients or both? > On Apr 15, 2019, at 11:37 AM, Alex McWhirter wrote: >> I went in and disabled TCP offload on all the nics, huge performance boost. >> went from 110MB/s to 240MB/s seq writes, reads lost a bit of performance >> going down to 680MB/s, but that's a decent trade off. Latency is still >> really high though, need to work on that. I think some more TCP tuning might >> help. >> >> > Those changes didn't do a whole lot, but i ended up enabling > performance.read-ahead on the gluster volume. my blockdev read ahead values > were already 8192, which seemed good enough. Not sure if ovirt set those, or > if it's just the defaults of my raid controller. > > Anyways up to 350MB/s writes, 700MB/s reads. Which so happens to correlate > with the saturation of my 10G network. Latency is still a slight issue, but > at least now im not blocking :) > > > ___ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-le...@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/5COPHAIVCVK42KMMGWZQVMNGDH6Q32ZC/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/T3QMRYHIDRZPUTW4QMGGVOCJ3S3VHLRY/
[ovirt-users] Re: Tuning Gluster Writes
On 2019-04-15 12:43, Darrell Budic wrote: > Interesting. Who's 10g cards and which offload settings did you disable? Did > you do that on the servers or the vm host clients or both? > > On Apr 15, 2019, at 11:37 AM, Alex McWhirter wrote: > > I went in and disabled TCP offload on all the nics, huge performance boost. > went from 110MB/s to 240MB/s seq writes, reads lost a bit of performance > going down to 680MB/s, but that's a decent trade off. Latency is still really > high though, need to work on that. I think some more TCP tuning might help. > > Those changes didn't do a whole lot, but i ended up enabling > performance.read-ahead on the gluster volume. my blockdev read ahead values > were already 8192, which seemed good enough. Not sure if ovirt set those, or > if it's just the defaults of my raid controller. > > Anyways up to 350MB/s writes, 700MB/s reads. Which so happens to correlate > with the saturation of my 10G network. Latency is still a slight issue, but > at least now im not blocking :) > > ___ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-le...@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/5COPHAIVCVK42KMMGWZQVMNGDH6Q32ZC/ These are dual port qlogic QLGE cards, plugging into dual Cisco Nexus 3064's with VPC to allow me to LACP across two switches. These are FCoE/10GBE cards, so on the Cisco Switches i had to disable lldp on the ports to stop FCoE initiator errors from disabling ports (as i don't use FCoE atm) bond options are "mode=4 lacp_rate=1 miimon=100 xmit_hash_policy=1" then i have the following /sbin/ifup-local script that triggers on storage network creation #!/bin/bash case "$1" in Storage) /sbin/ethtool -K ens2f0 tx off rx off tso off gso off /sbin/ethtool -K ens2f1 tx off rx off tso off gso off /sbin/ip link set dev ens2f0 txqueuelen 1 /sbin/ip link set dev ens2f1 txqueuelen 1 /sbin/ip link set dev bond2 txqueuelen 1 /sbin/ip link set dev Storage txqueuelen 1 ;; *) ;; esac exit 0 if you have lro, disable it too IMO, these cards do not do lro so it's not applicable to me. This did cut down my read performance by about 50MB/s, but my write went from 98-110MB/s to about 240MB/s, then enabling read-ahead got me to the 350MB/s it should have been.___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/Y6ZHB3JSQS6GLXWUWU6N7JBMRN5EUANS/
[ovirt-users] Re: Tuning Gluster Writes
On 2019-04-15 12:58, Alex McWhirter wrote: > On 2019-04-15 12:43, Darrell Budic wrote: Interesting. Who's 10g cards and > which offload settings did you disable? Did you do that on the servers or the > vm host clients or both? > > On Apr 15, 2019, at 11:37 AM, Alex McWhirter wrote: > > I went in and disabled TCP offload on all the nics, huge performance boost. > went from 110MB/s to 240MB/s seq writes, reads lost a bit of performance > going down to 680MB/s, but that's a decent trade off. Latency is still really > high though, need to work on that. I think some more TCP tuning might help. > > Those changes didn't do a whole lot, but i ended up enabling > performance.read-ahead on the gluster volume. my blockdev read ahead values > were already 8192, which seemed good enough. Not sure if ovirt set those, or > if it's just the defaults of my raid controller. > > Anyways up to 350MB/s writes, 700MB/s reads. Which so happens to correlate > with the saturation of my 10G network. Latency is still a slight issue, but > at least now im not blocking :) > > ___ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-le...@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/5COPHAIVCVK42KMMGWZQVMNGDH6Q32ZC/ These are dual port qlogic QLGE cards, plugging into dual Cisco Nexus 3064's with VPC to allow me to LACP across two switches. These are FCoE/10GBE cards, so on the Cisco Switches i had to disable lldp on the ports to stop FCoE initiator errors from disabling ports (as i don't use FCoE atm) bond options are "mode=4 lacp_rate=1 miimon=100 xmit_hash_policy=1" then i have the following /sbin/ifup-local script that triggers on storage network creation #!/bin/bash case "$1" in Storage) /sbin/ethtool -K ens2f0 tx off rx off tso off gso off /sbin/ethtool -K ens2f1 tx off rx off tso off gso off /sbin/ip link set dev ens2f0 txqueuelen 1 /sbin/ip link set dev ens2f1 txqueuelen 1 /sbin/ip link set dev bond2 txqueuelen 1 /sbin/ip link set dev Storage txqueuelen 1 ;; *) ;; esac exit 0 if you have lro, disable it too IMO, these cards do not do lro so it's not applicable to me. This did cut down my read performance by about 50MB/s, but my write went from 98-110MB/s to about 240MB/s, then enabling read-ahead got me to the 350MB/s it should have been. Oh and i did it on both, the VM hosts and storage machines. Same cards in all of them.___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/EZF2G3AWM3GSDR35CLP2SKABALQ6H4KN/
