Hi Jorick, Thank you for that, we will try direct-io-mode=disable.
Would anyone have any other advice on the cause of 114MBps of network traffic with GFS? On Fri, 20 Dec 2019 at 03:00, Jorick Astrego <jor...@netbulae.eu> wrote: > Hi David, > > Did you try setting "direct-io-mode=disable" on the client mounts? As it > is mostly static content it would help to use the kernel caching and > read-ahead mechanisms. > > I think the default is enabled. > > Regards, > > Jorick Astrego > On 12/19/19 1:28 AM, David Cunningham wrote: > > Hi Raghavendra and Strahil, > > We are using GFS version 5.6-1.el7 from the CentOS repository. > Unfortunately we can't modify the application and it expects to read and > write from a normal filesystem. > > There's around 25GB of data being written during a business day, so over > 10 hours that's around 0.7 MBps, which has me mystified as to how it can > generate 114MBps of network traffic. Granted we have read traffic as well, > but still. The chart shows much more inbound traffic to the GFS server than > outbound, suggesting the problem is with data writes. > > Is it possible with GFS to not check with the other nodes when reading? > Our data is mostly static and we don't require 100% guarantee that the data > is up-to-date when reading. > > Thanks for any assistance. > > > On Wed, 18 Dec 2019 at 16:39, Raghavendra Gowdappa <rgowd...@redhat.com> > wrote: > >> What version of Glusterfs are you using? Though, not sure what's the root >> cause of your problem, just wanted to point out a bug with read-ahead which >> would cause read-amplification over network [1][2], which should be fixed >> in recent versions. >> >> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1214489 >> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1393419 >> >> On Wed, Dec 18, 2019 at 2:50 AM David Cunningham < >> dcunning...@voisonics.com> wrote: >> >>> Hello, >>> >>> We switched a production system to using GFS instead of NFS at the >>> weekend, however it didn't go well on Monday when full load hit. The >>> application started crashing regularly and we had to revert to NFS. It >>> seems that the problem was high network traffic used by GFS. >>> >>> We've two GFS nodes plus one arbiter node, each about 1.3ms latency from >>> each other. Attached is a chart of network traffic on one of the GFS nodes. >>> We see that it saturated the 1Gbps link before we reverted to NFS at 15:10. >>> >>> The question is, why does GFS use so much network traffic and is there >>> anything we can do about it? NFS traffic doesn't exceed 4MBps, so 120MBps >>> for GFS seems awfully high. >>> >>> It would also be good to have faster read performance from GFS, but >>> that's another issue. >>> >>> Thanks in advance for any assistance. >>> >>> -- >>> David Cunningham, Voisonics Limited >>> http://voisonics.com/ >>> USA: +1 213 221 1092 >>> New Zealand: +64 (0)28 2558 3782 >>> ________ >>> >>> Community Meeting Calendar: >>> >>> APAC Schedule - >>> Every 2nd and 4th Tuesday at 11:30 AM IST >>> Bridge: https://bluejeans.com/441850968 >>> >>> NA/EMEA Schedule - >>> Every 1st and 3rd Tuesday at 01:00 PM EDT >>> Bridge: https://bluejeans.com/441850968 >>> >>> Gluster-users mailing list >>> Gluster-users@gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users >>> >> > > -- > David Cunningham, Voisonics Limited > http://voisonics.com/ > USA: +1 213 221 1092 > New Zealand: +64 (0)28 2558 3782 > > ________ > > Community Meeting Calendar: > > APAC Schedule - > Every 2nd and 4th Tuesday at 11:30 AM IST > Bridge: https://bluejeans.com/441850968 > > NA/EMEA Schedule - > Every 1st and 3rd Tuesday at 01:00 PM EDT > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing > listGluster-users@gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users > > > > > > Met vriendelijke groet, With kind regards, > > Jorick Astrego > > *Netbulae Virtualization Experts * > ------------------------------ > Tel: 053 20 30 270 i...@netbulae.eu Staalsteden 4-3A KvK 08198180 > Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW NL821234584B01 > ------------------------------ > > ________ > > Community Meeting Calendar: > > APAC Schedule - > Every 2nd and 4th Tuesday at 11:30 AM IST > Bridge: https://bluejeans.com/441850968 > > NA/EMEA Schedule - > Every 1st and 3rd Tuesday at 01:00 PM EDT > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > -- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782
________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users