Re: very poor nfs performance
On Sat, 2024-03-09 at 13:54 +0100, hw wrote: > > NFS can be hard on network card drivers > IPv6 may be faster than IPv4 > the network cable might suck > the switch might suck or block stuff As iperf and other network protocols were confirmed to be fast by the OP it is very unlikely that it is a straight network problem. Yes, these effects do exist occasionally (weird interactions of higher level protocols and the low level stuff), but it is very rare. The cable that is so specifically broken to slow down NFS but not scp might exist, but it is very unlikely. /ralph
Re: very poor nfs performance
On Thu, 2024-03-07 at 10:13 +0100, Stefan K wrote: > Hello guys, > > I hope someone can help me with my problem. > Our NFS performance ist very bad, like ~20MB/s, mountoption looks like that: Reading or writing, or both? Try testing with files on a different volume. > rw,relatime,sync,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,local_lock=none try IPv6 > [...] > Only one nfs client(debian 12) is connected via 10G, try a good 1GB network card > since we host also database files on the nfs share, bad idea > 'sync'-mountoption is important (more or less), but it should still > be much faster than 20MB/s I wouldn't dare to go without sync other than for making backups maybe. Without sync, you probably need to test with a larger file. > so can somebody tell me whats wrong or what should I change to speed > that up? Guesswork: NFS can be hard on network card drivers IPv6 may be faster than IPv4 the network cable might suck the switch might suck or block stuff ZFS might suck in combination with NFS network cards might happen to be in disadvantageous slots network cards can get too hot try Fedora instead of Debian (boot a live system on the server, configure NFS and see what happens when serving files from BTRFS) do you see any unusual system load while transferring files? do you need to run more NFS processes (increase their limit) are you running irqbalance? are you running numad if you're on a numa machine? what CPU governors are you using? do the i/o schedulers interfere? signature.asc Description: This is a digitally signed message part
Re: very poor nfs performance
Mike Kupfer wrote: > Stefan K wrote: > > > > Can you partition the files into 2 different shares? Put the database > > > files in one share and access them using "sync", and put the rest of the > > > files in a different share, with no "sync"? > > this could be a solution, but I want to understand why is it so slow and > > fix that > > It's inherent in how sync works. Over-the-wire calls are expensive. > NFS implementations try to get acceptable performance by extensive > caching, using asynchronous operations when possible, and by issuing a > smaller number of large RPCs (rather than a larger number of small > RPCs). The sync option defeats all of those mechanisms. It is also the case that databases absolutely need sync to work properly, so running them over NFS is a bad idea. At most, a sqlite DB can be OK -- because sqlite is single user. -dsr-
Re: Aw: Re: Re: very poor nfs performance
Stefan K wrote: > > Can you partition the files into 2 different shares? Put the database > > files in one share and access them using "sync", and put the rest of the > > files in a different share, with no "sync"? > this could be a solution, but I want to understand why is it so slow and fix > that It's inherent in how sync works. Over-the-wire calls are expensive. NFS implementations try to get acceptable performance by extensive caching, using asynchronous operations when possible, and by issuing a smaller number of large RPCs (rather than a larger number of small RPCs). The sync option defeats all of those mechanisms. mike
Re: very poor nfs performance
Stefan K wrote: > > Run the database on the machine that stores the files and perform > > database access remotely over the net instead. ? > > yes, but this doesn't resolve the performance issue with nfs But it removes your issue that forces you to use the sync option.
Aw: Re: very poor nfs performance
> Run the database on the machine that stores the files and perform > database access remotely over the net instead. ? yes, but this doesn't resolve the performance issue with nfs
Aw: Re: Re: very poor nfs performance
> Can you partition the files into 2 different shares? Put the database > files in one share and access them using "sync", and put the rest of the > files in a different share, with no "sync"? this could be a solution, but I want to understand why is it so slow and fix that
Re: very poor nfs performance
Stefan K wrote: > > You could try removing the "sync" option, just as an experiment, to > > see how much it is contributing to the slowdown. > > If I don't use sync I got around 300MB/s (tested with 600MB-file) .. > that's ok (far from great), but since there are database files on the > nfs it can cause file/database corruption, so we would like to use > sync option Run the database on the machine that stores the files and perform database access remotely over the net instead. ? > best regards > Stefan
Aw: Re: very poor nfs performance
> You could try removing the "sync" option, just as an experiment, to see > how much it is contributing to the slowdown. If I don't use sync I got around 300MB/s (tested with 600MB-file) .. that's ok (far from great), but since there are database files on the nfs it can cause file/database corruption, so we would like to use sync option best regards Stefan
Aw: Re: very poor nfs performance
> You could test with noatime if you don't need access times. > And perhaps with lazytime instead of relatime. Mountoptions are: type zfs (rw,xattr,noacl) I get you point, but when you look at my fio output, the performance is quiet good > Could you provide us > nfsstat -v server: nfsstat -v Server packet stats: packetsudptcptcpconn 509979521 0 510004972 2 Server rpc stats: calls badcalls badfmt badauthbadclnt 509971853 0 0 0 0 Server reply cache: hits misses nocache 0 0 509980028 Server io stats: read write 1587531840 3079615002 Server read ahead cache: size 0-10% 10-20% 20-30% 30-40% 40-50% 50-60% 60-70% 70-80% 80-90% 90-100%notfound 0 0 0 0 0 0 0 0 0 0 0 0 Server file handle cache: lookup anon ncachedir ncachenondir stale 0 0 0 0 0 Server nfs v4: null compound 2 0% 509976662 99% Server nfs v4 operations: op0-unused op1-unused op2-future access close 0 0% 0 0% 0 0% 5015903 0% 3091693 0% commit create delegpurge delegreturn getattr 3146340% 1498360% 0 0% 1615740 0% 390748077 20% getfhlink lock locktlocku 2573550 0% 0 0% 170% 0 0% 150% lookup lookup_root nverify open openattr 3931149 0% 0 0% 0 0% 3131045 0% 0 0% open_confopen_dgrdputfhputpubfh putrootfh 0 0% 3 0% 510522216 26% 0 0% 4 0% read readdir readlink remove rename 59976532 3% 4217910% 0 0% 4299650% 2445640% renewrestorefhsavefh secinfo setattr 0 0% 0 0% 5422310% 0 0% 8453240% setcltid setcltidconf verify writerellockowner 0 0% 0 0% 0 0% 404569758 21% 0 0% bc_ctl bind_connexchange_id create_ses destroy_ses 0 0% 0 0% 4 0% 2 0% 1 0% free_stateid getdirdeleg getdevinfo getdevlist layoutcommit 150% 0 0% 0 0% 0 0% 0 0% layoutgetlayoutreturn secinfononam sequence set_ssv 0 0% 0 0% 2 0% 509980018 26% 0 0% test_stateid want_deleg destroy_clid reclaim_comp allocate 100% 0 0% 1 0% 2 0% 164 0% copy copy_notify deallocate ioadvise layouterror 2976670% 0 0% 0 0% 0 0% 0 0% layoutstats offloadcanceloffloadstatusreadplus seek 0 0% 0 0% 0 0% 0 0% 0 0% write_same 0 0% client: nfsstat -v Client packet stats: packetsudptcptcpconn 0 0 0 0 Client rpc stats: calls retransauthrefrsh 37415730 0 37425651 Client nfs v4: null read writecommit open 1 0% 4107833 10% 30388717 81% 2516 0% 55493 0% open_confopen_noatopen_dgrdclosesetattr 0 0% 1942520% 0 0% 2473800% 75890 0% fsinfo renewsetclntidconfirm lock 459 0% 0 0% 0 0% 0 0% 4 0% locktlockuaccess getattr lookup 0 0% 2 0% 1315330% 1497029 4% 3180560% lookup_root remove rename link symlink 1 0% 31656 0% 15877 0% 0 0% 0 0% create pathconf statfs readlink readdir 7019 0% 458 0% 1705220% 0 0% 30007 0% server_caps delegreturn getacl setacl fs_locations 917 0% 1181090% 0 0% 0 0% 0 0% rel_lkowner secinfo fsid_present exchange_id create_session 0 0% 0 0% 0 0% 2 0% 1 0% destroy_session sequence get_lease_time reclaim_comp layoutget 0 0% 0 0% 0
Re: very poor nfs performance
On 2024-03-07, Stefan K wrote: > I hope someone can help me with my problem. > Our NFS performance ist very bad, like ~20MB/s, mountoption looks like that: > rw,relatime,sync,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,local_lock=none What are the mount options for your zfs filesystem ? You could test with noatime if you don't need access times. And perhaps with lazytime instead of relatime. Could you provide us nfsstat -v
Re: very poor nfs performance
Stefan K wrote: > 'sync'-mountoption is important (more or less), but it should still be > much faster than 20MB/s I don't know if "sync" could be entirely responsible for such a slowdown, but it's likely at least contributing, particularly if the application is doing small I/Os at the system call level. You could try removing the "sync" option, just as an experiment, to see how much it is contributing to the slowdown. mike
Aw: Re: very poor nfs performance
Hi Ralph, I just tested it with scp and I got 262MB/s So it's not a network issue, just a NFS issue, somehow. best regards Stefan > Gesendet: Donnerstag, 07. März 2024 um 11:22 Uhr > Von: "Ralph Aichinger" > An: debian-user@lists.debian.org > Betreff: Re: very poor nfs performance > > On Thu, 2024-03-07 at 10:13 +0100, Stefan K wrote: > > Hello guys, > > > > I hope someone can help me with my problem. > > Our NFS performance ist very bad, like ~20MB/s, mountoption looks > > like that: > > Are both sides agreeing on MTU (using Jumbo frames or not)? > > Have you tested the network with iperf (or simiar), does this happen > only with NFS or also with other network traffic? > > /ralph > >
Re: very poor nfs performance
On Thu, 2024-03-07 at 10:13 +0100, Stefan K wrote: > Hello guys, > > I hope someone can help me with my problem. > Our NFS performance ist very bad, like ~20MB/s, mountoption looks > like that: Are both sides agreeing on MTU (using Jumbo frames or not)? Have you tested the network with iperf (or simiar), does this happen only with NFS or also with other network traffic? /ralph
very poor nfs performance
Hello guys, I hope someone can help me with my problem. Our NFS performance ist very bad, like ~20MB/s, mountoption looks like that: rw,relatime,sync,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,local_lock=none The NFS server (debian 12) is a ZFS Fileserver with 40G network connection, also the read/write performance is good: fio --rw=readwrite --rwmixread=70 --name=test --size=50G --direct=1 --bsrange=4k-1M --numjobs=30 --group_reporting --filename=/zfs_storage/test/asdfg --runtime=600 --time_based READ: bw=11.1GiB/s (11.9GB/s), 11.1GiB/s-11.1GiB/s (11.9GB/s-11.9GB/s), io=6665GiB (7156GB), run=64-64msec WRITE: bw=4875MiB/s (5112MB/s), 4875MiB/s-4875MiB/s (5112MB/s-5112MB/s), io=2856GiB (3067GB), run=64-64msec Only one nfs client(debian 12) is connected via 10G, since we host also database files on the nfs share, 'sync'-mountoption is important (more or less), but it should still be much faster than 20MB/s so can somebody tell me whats wrong or what should I change to speed that up? thanks in advance. best regards Stefan
