Hi David, Did you try setting "direct-io-mode=disable" on the client mounts? As it is mostly static content it would help to use the kernel caching and read-ahead mechanisms.
I think the default is enabled. Regards, Jorick Astrego On 12/19/19 1:28 AM, David Cunningham wrote: > Hi Raghavendra and Strahil, > > We are using GFS version 5.6-1.el7 from the CentOS repository. > Unfortunately we can't modify the application and it expects to read > and write from a normal filesystem. > > There's around 25GB of data being written during a business day, so > over 10 hours that's around 0.7 MBps, which has me mystified as to how > it can generate 114MBps of network traffic. Granted we have read > traffic as well, but still. The chart shows much more inbound traffic > to the GFS server than outbound, suggesting the problem is with data > writes. > > Is it possible with GFS to not check with the other nodes when > reading? Our data is mostly static and we don't require 100% guarantee > that the data is up-to-date when reading. > > Thanks for any assistance. > > > On Wed, 18 Dec 2019 at 16:39, Raghavendra Gowdappa > <rgowd...@redhat.com <mailto:rgowd...@redhat.com>> wrote: > > What version of Glusterfs are you using? Though, not sure what's > the root cause of your problem, just wanted to point out a bug > with read-ahead which would cause read-amplification over network > [1][2], which should be fixed in recent versions. > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1214489 > [2] https://bugzilla.redhat.com/show_bug.cgi?id=1393419 > > On Wed, Dec 18, 2019 at 2:50 AM David Cunningham > <dcunning...@voisonics.com <mailto:dcunning...@voisonics.com>> wrote: > > Hello, > > We switched a production system to using GFS instead of NFS at > the weekend, however it didn't go well on Monday when full > load hit. The application started crashing regularly and we > had to revert to NFS. It seems that the problem was high > network traffic used by GFS. > > We've two GFS nodes plus one arbiter node, each about 1.3ms > latency from each other. Attached is a chart of network > traffic on one of the GFS nodes. We see that it saturated the > 1Gbps link before we reverted to NFS at 15:10. > > The question is, why does GFS use so much network traffic and > is there anything we can do about it? NFS traffic doesn't > exceed 4MBps, so 120MBps for GFS seems awfully high. > > It would also be good to have faster read performance from > GFS, but that's another issue. > > Thanks in advance for any assistance. > > -- > David Cunningham, Voisonics Limited > http://voisonics.com/ > USA: +1 213 221 1092 > New Zealand: +64 (0)28 2558 3782 > ________ > > Community Meeting Calendar: > > APAC Schedule - > Every 2nd and 4th Tuesday at 11:30 AM IST > Bridge: https://bluejeans.com/441850968 > > NA/EMEA Schedule - > Every 1st and 3rd Tuesday at 01:00 PM EDT > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users@gluster.org <mailto:Gluster-users@gluster.org> > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > -- > David Cunningham, Voisonics Limited > http://voisonics.com/ > USA: +1 213 221 1092 > New Zealand: +64 (0)28 2558 3782 > > ________ > > Community Meeting Calendar: > > APAC Schedule - > Every 2nd and 4th Tuesday at 11:30 AM IST > Bridge: https://bluejeans.com/441850968 > > NA/EMEA Schedule - > Every 1st and 3rd Tuesday at 01:00 PM EDT > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users Met vriendelijke groet, With kind regards, Jorick Astrego Netbulae Virtualization Experts ---------------- Tel: 053 20 30 270 i...@netbulae.eu Staalsteden 4-3A KvK 08198180 Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW NL821234584B01 ----------------
________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users