Here is the output for our ssd gluster which exhibits the same issue as the hdd glusters. However, I can replicate the issue on an 8TB WD Gold disk NFS mounted as well ( removed the gluster part ) Which is the reason I'm on the oVirt site. I can start a file copy that writes at max speed, then after a gb or 2 it drops down to 3-10 MBps maxing at 13.3 ish overall. Testing outside of oVirt using dd doesn't have the same behavior. Outside ovirt (directly on the ovirtnode to the gluster or 8tb nfs mount results is max drive speeds consistently for large file copies)
I enabled writeback (as someone suggested) on the virtio-scsi windows disk and one of our windows 10 installs speed up. Still suffers from sustained write issue which causes the whole box to cripple. Opening chrome for example cripples the box or sql server management studio also. Volume Name: gv1 Type: Replicate Volume ID: 7340a436-d971-4d69-84f9-12a23cd76ec8 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: 10.30.30.121:/gluster_bricks/gv1/brick Brick2: 10.30.30.122:/gluster_bricks/gv1/brick Brick3: 10.30.30.123:/gluster_bricks/gv1/brick (arbiter) Options Reconfigured: network.ping-timeout: 30 performance.strict-o-direct: on storage.owner-gid: 36 storage.owner-uid: 36 user.cifs: off cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off features.shard: off cluster.granular-entry-heal: enable transport.address-family: inet nfs.disable: on performance.client-io-threads: off On Thu, Mar 7, 2019 at 1:00 AM Krutika Dhananjay <kdhan...@redhat.com> wrote: > So from the profile, it appears the XATTROPs and FINODELKs are way higher > than the number of WRITEs: > > <excerpt> > ... > ... > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.43 384.83 us 51.00 us 65375.00 us 13632 > FXATTROP > 7.54 13535.70 us 225.00 us 210298.00 us 6816 > WRITE > 45.99 28508.86 us 7.00 us 2591280.00 us 19751 > FINODELK > > ... > ... > </excerpt> > > We'd noticed something similar in our internal tests and found > inefficiencies in gluster's eager-lock implementation. This was fixed at > https://review.gluster.org/c/glusterfs/+/19503. > I need the two things I asked for in the prev mail to confirm if you're > hitting the same issue. > > -Krutika > > On Thu, Mar 7, 2019 at 12:24 PM Krutika Dhananjay <kdhan...@redhat.com> > wrote: > >> Hi, >> >> Could you share the following pieces of information to begin with - >> >> 1. output of `gluster volume info $AFFECTED_VOLUME_NAME` >> 2. glusterfs version you're running >> >> -Krutika >> >> >> On Sat, Mar 2, 2019 at 3:38 AM Drew R <drew.r...@gmail.com> wrote: >> >>> Saw some people asking for profile info. So I had started a migration >>> from a 6TB WDGold 2+1arb replicated gluster to a 1TB samsung ssd 2+1 rep >>> gluster and it's been running a while for a 100GB file thin provisioned >>> with like 28GB actually used. Here is the profile info. I started the >>> profiler like 5 minutes ago. The migration had been running for like >>> 30minutes: >>> >>> gluster volume profile gv2 info >>> Brick: 10.30.30.122:/gluster_bricks/gv2/brick >>> --------------------------------------------- >>> Cumulative Stats: >>> Block Size: 256b+ 512b+ >>> 1024b+ >>> No. of Reads: 1189 8 >>> 12 >>> No. of Writes: 4 3245 >>> 883 >>> >>> Block Size: 2048b+ 4096b+ >>> 8192b+ >>> No. of Reads: 10 20 >>> 2 >>> No. of Writes: 1087 312228 >>> 124080 >>> >>> Block Size: 16384b+ 32768b+ >>> 65536b+ >>> No. of Reads: 0 1 >>> 52 >>> No. of Writes: 5188 3617 >>> 5532 >>> >>> Block Size: 131072b+ >>> No. of Reads: 70191 >>> No. of Writes: 634192 >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls >>> Fop >>> --------- ----------- ----------- ----------- ------------ >>> ---- >>> 0.00 0.00 us 0.00 us 0.00 us 2 >>> FORGET >>> 0.00 0.00 us 0.00 us 0.00 us 202 >>> RELEASE >>> 0.00 0.00 us 0.00 us 0.00 us 1297 >>> RELEASEDIR >>> 0.00 14.50 us 9.00 us 20.00 us 4 >>> READDIR >>> 0.00 38.00 us 9.00 us 120.00 us 7 >>> GETXATTR >>> 0.00 66.00 us 34.00 us 128.00 us 6 >>> OPEN >>> 0.00 137.25 us 52.00 us 195.00 us 4 >>> SETATTR >>> 0.00 23.19 us 11.00 us 46.00 us 26 >>> INODELK >>> 0.00 41.58 us 18.00 us 79.00 us 24 >>> OPENDIR >>> 0.00 166.70 us 15.00 us 775.00 us 27 >>> READDIRP >>> 0.01 135.29 us 12.00 us 11695.00 us 221 >>> STATFS >>> 0.01 176.54 us 22.00 us 22944.00 us 364 >>> FSTAT >>> 0.02 626.21 us 13.00 us 17308.00 us 168 >>> STAT >>> 0.09 834.84 us 9.00 us 34337.00 us 607 >>> LOOKUP >>> 0.73 146.18 us 6.00 us 52255.00 us 29329 >>> FINODELK >>> 1.03 298.20 us 42.00 us 43711.00 us 20204 >>> FXATTROP >>> 15.38 8903.40 us 213.00 us 213832.00 us 10102 >>> WRITE >>> 39.14 26796.37 us 222.00 us 122696.00 us 8538 >>> READ >>> 43.59 39536.79 us 259.00 us 183630.00 us 6446 >>> FSYNC >>> >>> Duration: 15078 seconds >>> Data Read: 9207377205 bytes >>> Data Written: 86214017762 bytes >>> >>> Interval 2 Stats: >>> Block Size: 256b+ 512b+ >>> 1024b+ >>> No. of Reads: 17 0 >>> 0 >>> No. of Writes: 0 43 >>> 7 >>> >>> Block Size: 2048b+ 4096b+ >>> 8192b+ >>> No. of Reads: 0 7 >>> 0 >>> No. of Writes: 16 1881 >>> 1010 >>> >>> Block Size: 16384b+ 32768b+ >>> 65536b+ >>> No. of Reads: 0 0 >>> 6 >>> No. of Writes: 305 586 >>> 2359 >>> >>> Block Size: 131072b+ >>> No. of Reads: 7162 >>> No. of Writes: 610 >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls >>> Fop >>> --------- ----------- ----------- ----------- ------------ >>> ---- >>> 0.00 0.00 us 0.00 us 0.00 us 6 >>> RELEASE >>> 0.00 0.00 us 0.00 us 0.00 us 20 >>> RELEASEDIR >>> 0.00 14.50 us 9.00 us 20.00 us 4 >>> READDIR >>> 0.00 38.00 us 9.00 us 120.00 us 7 >>> GETXATTR >>> 0.00 66.00 us 34.00 us 128.00 us 6 >>> OPEN >>> 0.00 137.25 us 52.00 us 195.00 us 4 >>> SETATTR >>> 0.00 23.19 us 11.00 us 46.00 us 26 >>> INODELK >>> 0.00 40.05 us 18.00 us 79.00 us 20 >>> OPENDIR >>> 0.00 180.33 us 16.00 us 775.00 us 21 >>> READDIRP >>> 0.01 181.77 us 12.00 us 11695.00 us 149 >>> STATFS >>> 0.01 511.23 us 15.00 us 16954.00 us 111 >>> STAT >>> 0.01 210.22 us 22.00 us 22944.00 us 291 >>> FSTAT >>> 0.09 979.73 us 9.00 us 23539.00 us 404 >>> LOOKUP >>> 0.81 179.58 us 6.00 us 52255.00 us 19786 >>> FINODELK >>> 1.07 342.71 us 42.00 us 43711.00 us 13633 >>> FXATTROP >>> 14.18 9087.30 us 213.00 us 213832.00 us 6817 >>> WRITE >>> 40.00 40227.38 us 259.00 us 157934.00 us 4343 >>> FSYNC >>> 43.81 26601.57 us 222.00 us 118756.00 us 7192 >>> READ >>> >>> Duration: 177 seconds >>> Data Read: 939534945 bytes >>> Data Written: 363763200 bytes >>> >>> Brick: 10.30.30.121:/gluster_bricks/gv2/brick >>> --------------------------------------------- >>> Cumulative Stats: >>> Block Size: 8b+ 256b+ >>> 512b+ >>> No. of Reads: 0 142626 >>> 329092 >>> No. of Writes: 12 98 >>> 123230 >>> >>> Block Size: 1024b+ 2048b+ >>> 4096b+ >>> No. of Reads: 21175 18223 >>> 3345043 >>> No. of Writes: 58646 69147 >>> 6092143 >>> >>> Block Size: 8192b+ 16384b+ >>> 32768b+ >>> No. of Reads: 1090937 216037 >>> 748595 >>> No. of Writes: 2030403 243915 >>> 240942 >>> >>> Block Size: 65536b+ 131072b+ >>> No. of Reads: 1524667 4828662 >>> No. of Writes: 605125 1889892 >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls >>> Fop >>> --------- ----------- ----------- ----------- ------------ >>> ---- >>> 0.00 0.00 us 0.00 us 0.00 us 133 >>> FORGET >>> 0.00 0.00 us 0.00 us 0.00 us 4821 >>> RELEASE >>> 0.00 0.00 us 0.00 us 0.00 us 42742 >>> RELEASEDIR >>> 0.00 28.50 us 12.00 us 61.00 us 4 >>> READDIR >>> 0.00 71.75 us 59.00 us 88.00 us 4 >>> SETATTR >>> 0.00 44.57 us 12.00 us 94.00 us 7 >>> GETXATTR >>> 0.00 62.17 us 42.00 us 89.00 us 6 >>> OPEN >>> 0.00 35.92 us 1.00 us 76.00 us 24 >>> OPENDIR >>> 0.00 325.54 us 22.00 us 1117.00 us 13 >>> READDIRP >>> 0.00 482.13 us 19.00 us 12473.00 us 46 >>> STAT >>> 0.00 217.65 us 15.00 us 32841.00 us 221 >>> STATFS >>> 0.01 497.52 us 11.00 us 29179.00 us 539 >>> LOOKUP >>> 0.02 685.44 us 16.00 us 61131.00 us 615 >>> FSTAT >>> 0.04 31691.92 us 17.00 us 349648.00 us 26 >>> INODELK >>> 0.45 451.95 us 51.00 us 65375.00 us 20206 >>> FXATTROP >>> 6.90 13906.03 us 225.00 us 210298.00 us 10103 >>> WRITE >>> 18.47 58375.96 us 389.00 us 218211.00 us 6446 >>> FSYNC >>> 27.45 27279.84 us 184.00 us 241444.00 us 20503 >>> READ >>> 46.66 32462.82 us 7.00 us 2750519.00 us 29283 >>> FINODELK >>> >>> Duration: 475152 seconds >>> Data Read: 799630169437 bytes >>> Data Written: 362708560796 bytes >>> >>> Interval 2 Stats: >>> Block Size: 256b+ 512b+ >>> 1024b+ >>> No. of Reads: 53 0 >>> 0 >>> No. of Writes: 0 43 >>> 7 >>> >>> Block Size: 2048b+ 4096b+ >>> 8192b+ >>> No. of Reads: 0 891 >>> 616 >>> No. of Writes: 16 1881 >>> 1010 >>> >>> Block Size: 16384b+ 32768b+ >>> 65536b+ >>> No. of Reads: 288 750 >>> 52 >>> No. of Writes: 304 586 >>> 2359 >>> >>> Block Size: 131072b+ >>> No. of Reads: 9445 >>> No. of Writes: 610 >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls >>> Fop >>> --------- ----------- ----------- ----------- ------------ >>> ---- >>> 0.00 0.00 us 0.00 us 0.00 us 6 >>> RELEASE >>> 0.00 0.00 us 0.00 us 0.00 us 20 >>> RELEASEDIR >>> 0.00 28.50 us 12.00 us 61.00 us 4 >>> READDIR >>> 0.00 71.75 us 59.00 us 88.00 us 4 >>> SETATTR >>> 0.00 44.57 us 12.00 us 94.00 us 7 >>> GETXATTR >>> 0.00 62.17 us 42.00 us 89.00 us 6 >>> OPEN >>> 0.00 34.50 us 1.00 us 76.00 us 20 >>> OPENDIR >>> 0.00 55.16 us 19.00 us 128.00 us 25 >>> STAT >>> 0.00 363.18 us 25.00 us 1117.00 us 11 >>> READDIRP >>> 0.00 71.61 us 15.00 us 4323.00 us 149 >>> STATFS >>> 0.01 251.40 us 11.00 us 15344.00 us 336 >>> LOOKUP >>> 0.02 594.11 us 16.00 us 61131.00 us 450 >>> FSTAT >>> 0.07 31691.92 us 17.00 us 349648.00 us 26 >>> INODELK >>> 0.43 384.83 us 51.00 us 65375.00 us 13632 >>> FXATTROP >>> 7.54 13535.70 us 225.00 us 210298.00 us 6816 >>> WRITE >>> 20.03 56497.17 us 394.00 us 218211.00 us 4340 >>> FSYNC >>> 25.91 26229.86 us 184.00 us 241444.00 us 12095 >>> READ >>> 45.99 28508.86 us 7.00 us 2591280.00 us 19751 >>> FINODELK >>> >>> Duration: 177 seconds >>> Data Read: 1298631621 bytes >>> Data Written: 363744256 bytes >>> >>> Brick: 10.30.30.123:/gluster_bricks/gv2/brick >>> --------------------------------------------- >>> Cumulative Stats: >>> Block Size: 1b+ >>> No. of Reads: 0 >>> No. of Writes: 18126558 >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls >>> Fop >>> --------- ----------- ----------- ----------- ------------ >>> ---- >>> 0.00 0.00 us 0.00 us 0.00 us 383 >>> FORGET >>> 0.00 0.00 us 0.00 us 0.00 us 9602 >>> RELEASE >>> 0.00 0.00 us 0.00 us 0.00 us 70644 >>> RELEASEDIR >>> 0.00 20.75 us 13.00 us 29.00 us 4 >>> READDIR >>> 0.00 87.00 us 72.00 us 129.00 us 4 >>> SETATTR >>> 0.00 71.00 us 46.00 us 93.00 us 6 >>> OPEN >>> 0.00 66.00 us 9.00 us 114.00 us 7 >>> GETXATTR >>> 0.00 50.58 us 16.00 us 90.00 us 26 >>> INODELK >>> 0.00 56.38 us 1.00 us 111.00 us 24 >>> OPENDIR >>> 0.02 66.23 us 28.00 us 146.00 us 113 >>> FSTAT >>> 0.17 143.88 us 13.00 us 444.00 us 539 >>> LOOKUP >>> 1.17 51.64 us 10.00 us 166.00 us 10103 >>> WRITE >>> 2.89 44.06 us 7.00 us 2177.00 us 29349 >>> FINODELK >>> 5.55 122.83 us 54.00 us 9281.00 us 20206 >>> FXATTROP >>> 90.19 6249.64 us 237.00 us 59996.00 us 6448 >>> FSYNC >>> >>> Duration: 772792 seconds >>> Data Read: 0 bytes >>> Data Written: 18126558 bytes >>> >>> Interval 2 Stats: >>> Block Size: 1b+ >>> No. of Reads: 0 >>> No. of Writes: 6816 >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls >>> Fop >>> --------- ----------- ----------- ----------- ------------ >>> ---- >>> 0.00 0.00 us 0.00 us 0.00 us 6 >>> RELEASE >>> 0.00 0.00 us 0.00 us 0.00 us 20 >>> RELEASEDIR >>> 0.00 20.75 us 13.00 us 29.00 us 4 >>> READDIR >>> 0.00 87.00 us 72.00 us 129.00 us 4 >>> SETATTR >>> 0.00 71.00 us 46.00 us 93.00 us 6 >>> OPEN >>> 0.00 66.00 us 9.00 us 114.00 us 7 >>> GETXATTR >>> 0.00 58.10 us 1.00 us 111.00 us 20 >>> OPENDIR >>> 0.00 50.58 us 16.00 us 90.00 us 26 >>> INODELK >>> 0.03 66.23 us 28.00 us 146.00 us 113 >>> FSTAT >>> 0.16 145.84 us 13.00 us 346.00 us 336 >>> LOOKUP >>> 1.18 51.98 us 11.00 us 157.00 us 6816 >>> WRITE >>> 2.91 44.05 us 7.00 us 2177.00 us 19798 >>> FINODELK >>> 5.59 122.70 us 54.00 us 9281.00 us 13632 >>> FXATTROP >>> 90.11 6210.61 us 238.00 us 54711.00 us 4342 >>> FSYNC >>> >>> Duration: 177 seconds >>> Data Read: 0 bytes >>> Data Written: 6816 bytes >>> _______________________________________________ >>> Users mailing list -- users@ovirt.org >>> To unsubscribe send an email to users-le...@ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/XHAUIMGRCVEDCLS7F6WALUWJVQ4IXAWU/ >>> >>
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/BY7GYK3URRQQ5V4Y4GJ6M5ILLSXK2DNG/
