Re: bad NFS/UDP performance
> > On Sat, 4 Oct 2008, Danny Braniss wrote: > > > at the moment, the best I can do is run it on a different hardware that has > > if_em, the results are in > > ftp://ftp.cs.huji.ac.il/users/danny/lock.prof/7.1-1000.em the > > benchmark ran better with the Intel NIC, averaged UDP 54MB/s, TCP 53MB/s (I > > get the same numbers with an older kernel). > > Dear Danny: > > Unfortunately, I was left slightly unclear on the comparison you are making > above. Could you confirm whether or not, with if_em, you see a performance > regression using UDP NFS between 7.0-RELEASE and the most recent 7.1-STABLE, > and if you do, whether or not the RLOCK->WLOCK change has any effect on > performance? It would be nice to know on the same hardware but at least with > different hardware we get a sense of whether or not this might affect other > systems or whether it's limited to a narrower set of configurations. > > Thanks, 7.1-1000.em vanilla 7.1 1 x Intel Core Duo 7.1-1000.x2200.em vanilla 7.1 2 x Dual-Core AMD Opteron 7.0-1000.x2200.em 7.0 + RLOCK->WLOCK the plot thickens. I put an em card in, and the throughput is almost the same than with the bge. all the tests were done on the same host, a Sun x2200/amd/2cpux2core except for the one over the weekend that is a intel Core Duo, and not the same if_em card, sorry about that but one has PCI X, the other PCI Express :-(. what is becoming obvious is that NFS/UDP is very temperamental/sensitive :-) danny ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
On Sat, 4 Oct 2008, Danny Braniss wrote: at the moment, the best I can do is run it on a different hardware that has if_em, the results are in ftp://ftp.cs.huji.ac.il/users/danny/lock.prof/7.1-1000.em the benchmark ran better with the Intel NIC, averaged UDP 54MB/s, TCP 53MB/s (I get the same numbers with an older kernel). Dear Danny: Unfortunately, I was left slightly unclear on the comparison you are making above. Could you confirm whether or not, with if_em, you see a performance regression using UDP NFS between 7.0-RELEASE and the most recent 7.1-STABLE, and if you do, whether or not the RLOCK->WLOCK change has any effect on performance? It would be nice to know on the same hardware but at least with different hardware we get a sense of whether or not this might affect other systems or whether it's limited to a narrower set of configurations. Thanks, Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
>
> On Fri, 3 Oct 2008, Danny Braniss wrote:
>
> >> On Fri, 3 Oct 2008, Danny Braniss wrote:
> >>
> >>> gladly, but have no idea how to do LOCK_PROFILING, so some pointers would
> >>> be helpfull.
> >>
> >> The LOCK_PROFILING(9) man page isn't a bad starting point -- I find that
> >> the defaults work fine most of the time, so just use them. Turn the
> >> enable
> >> syscl on just before you begin a run, and turn it off immediately
> >> afterwards. Make sure to reset between reruns (rebooting to a new kernel
> >> is fine too!).
> >
> > in ftp://ftp.cs.huji.ac.il/users/danny/lock.prof
> > there 3 files:
> > 7.1-100 host connected at 100 running -prerelease
> > 7.1-1000same but connected at 1000
> > 7.0-1000-stable with your 'patch'
> > at 100 my benchmark didn't suffer from the profiling, average was about 9.
> > at 1000 the benchmark got realy hit, average was around 12 for the patched,
> > and 4 for the unpatched (less than at 100).
>
> Interesting. A bit of post-processing:
>
> [EMAIL PROTECTED]:/tmp> cat 7.1-1000 | awk -F' ' '{print $3" "$9}' | sort -n
> |
> tail -10
> 2413283 /r+d/7/sys/kern/kern_mutex.c:141
> 2470096 /r+d/7/sys/nfsclient/nfs_socket.c:1218
> 2676282 /r+d/7/sys/net/route.c:293
> 2754866 /r+d/7/sys/kern/vfs_bio.c:1468
> 3196298 /r+d/7/sys/nfsclient/nfs_bio.c:1664
> 3318742 /r+d/7/sys/net/route.c:1584
> 3711139 /r+d/7/sys/dev/bge/if_bge.c:3287
> 3753518 /r+d/7/sys/net/if_ethersubr.c:405
> 3961312 /r+d/7/sys/nfsclient/nfs_subs.c:1066
> 10688531 /r+d/7/sys/dev/bge/if_bge.c:3726
> [EMAIL PROTECTED]:/tmp> cat 7.0-1000 | awk -F' ' '{print $3" "$9}' | sort -n
> |
> tail -10
> 468631 /r+d/hunt/src/sys/nfsclient/nfs_nfsiod.c:286
> 501989 /r+d/hunt/src/sys/nfsclient/nfs_vnops.c:1148
> 631587 /r+d/hunt/src/sys/nfsclient/nfs_socket.c:1198
> 701155 /r+d/hunt/src/sys/nfsclient/nfs_socket.c:1258
> 718211 /r+d/hunt/src/sys/kern/kern_mutex.c:141
> 1118711 /r+d/hunt/src/sys/nfsclient/nfs_bio.c:1664
> 1169125 /r+d/hunt/src/sys/nfsclient/nfs_subs.c:1066
> 1222867 /r+d/hunt/src/sys/kern/vfs_bio.c:1468
> 3876072 /r+d/hunt/src/sys/netinet/udp_usrreq.c:545
> 5198927 /r+d/hunt/src/sys/netinet/udp_usrreq.c:864
>
> The first set above is with the unmodified 7-STABLE tree, the second with a
> reversion of read locking on the UDP inpcb. The big blinking sign of
> interest
> is that the bge interface lock is massively contended in the first set of
> output, and basically doesn't appear in the second. There are various
> reasons
> bge could stand out quite so much -- one possibly is that previously, the udp
> lock serialized all access to the interface from the send code, preventing
> the
> send and receive paths from contending.
>
> A few things to try:
>
> - Let's look compare the context switch rates on the two benchmarks. Could
>you run vmstat and look at the cpu cs line during the benchmarks and see
> how
>similar the two are as the benchmarks run? You'll want to run it with
>vmstat -w 1 and collect several samples per benchmark, since we're really
>interested in the distribution rather than an individual sample.
>
> - Is there any chance you could drop an if_em card into the same box and run
>the identical benchmarks with and without LOCK_PROFILING to see whether it
>behaves differently than bge when the patch is applied? if_em's interrupt
>handling is quite different, and may significantly affect lock use, and
>hence contention.
at the moment, the best I can do is run it on a different hardware that has
if_em,
the results are in
ftp://ftp.cs.huji.ac.il/users/danny/lock.prof/7.1-1000.em
the benchmark ran better with the Intel NIC, averaged UDP 54MB/s, TCP 53MB/s
(I get the same numbers with an older kernel).
danny
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
On Fri, 3 Oct 2008, Danny Braniss wrote:
On Fri, 3 Oct 2008, Danny Braniss wrote:
gladly, but have no idea how to do LOCK_PROFILING, so some pointers would
be helpfull.
The LOCK_PROFILING(9) man page isn't a bad starting point -- I find that
the defaults work fine most of the time, so just use them. Turn the enable
syscl on just before you begin a run, and turn it off immediately
afterwards. Make sure to reset between reruns (rebooting to a new kernel
is fine too!).
in ftp://ftp.cs.huji.ac.il/users/danny/lock.prof
there 3 files:
7.1-100 host connected at 100 running -prerelease
7.1-1000same but connected at 1000
7.0-1000-stable with your 'patch'
at 100 my benchmark didn't suffer from the profiling, average was about 9.
at 1000 the benchmark got realy hit, average was around 12 for the patched,
and 4 for the unpatched (less than at 100).
Interesting. A bit of post-processing:
[EMAIL PROTECTED]:/tmp> cat 7.1-1000 | awk -F' ' '{print $3" "$9}' | sort -n |
tail -10
2413283 /r+d/7/sys/kern/kern_mutex.c:141
2470096 /r+d/7/sys/nfsclient/nfs_socket.c:1218
2676282 /r+d/7/sys/net/route.c:293
2754866 /r+d/7/sys/kern/vfs_bio.c:1468
3196298 /r+d/7/sys/nfsclient/nfs_bio.c:1664
3318742 /r+d/7/sys/net/route.c:1584
3711139 /r+d/7/sys/dev/bge/if_bge.c:3287
3753518 /r+d/7/sys/net/if_ethersubr.c:405
3961312 /r+d/7/sys/nfsclient/nfs_subs.c:1066
10688531 /r+d/7/sys/dev/bge/if_bge.c:3726
[EMAIL PROTECTED]:/tmp> cat 7.0-1000 | awk -F' ' '{print $3" "$9}' | sort -n |
tail -10
468631 /r+d/hunt/src/sys/nfsclient/nfs_nfsiod.c:286
501989 /r+d/hunt/src/sys/nfsclient/nfs_vnops.c:1148
631587 /r+d/hunt/src/sys/nfsclient/nfs_socket.c:1198
701155 /r+d/hunt/src/sys/nfsclient/nfs_socket.c:1258
718211 /r+d/hunt/src/sys/kern/kern_mutex.c:141
1118711 /r+d/hunt/src/sys/nfsclient/nfs_bio.c:1664
1169125 /r+d/hunt/src/sys/nfsclient/nfs_subs.c:1066
1222867 /r+d/hunt/src/sys/kern/vfs_bio.c:1468
3876072 /r+d/hunt/src/sys/netinet/udp_usrreq.c:545
5198927 /r+d/hunt/src/sys/netinet/udp_usrreq.c:864
The first set above is with the unmodified 7-STABLE tree, the second with a
reversion of read locking on the UDP inpcb. The big blinking sign of interest
is that the bge interface lock is massively contended in the first set of
output, and basically doesn't appear in the second. There are various reasons
bge could stand out quite so much -- one possibly is that previously, the udp
lock serialized all access to the interface from the send code, preventing the
send and receive paths from contending.
A few things to try:
- Let's look compare the context switch rates on the two benchmarks. Could
you run vmstat and look at the cpu cs line during the benchmarks and see how
similar the two are as the benchmarks run? You'll want to run it with
vmstat -w 1 and collect several samples per benchmark, since we're really
interested in the distribution rather than an individual sample.
- Is there any chance you could drop an if_em card into the same box and run
the identical benchmarks with and without LOCK_PROFILING to see whether it
behaves differently than bge when the patch is applied? if_em's interrupt
handling is quite different, and may significantly affect lock use, and
hence contention.
Thanks,
Robert N M Watson
Computer Laboratory
University of Cambridge
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
> > On Fri, 3 Oct 2008, Danny Braniss wrote: > > > gladly, but have no idea how to do LOCK_PROFILING, so some pointers would > > be > > helpfull. > > The LOCK_PROFILING(9) man page isn't a bad starting point -- I find that the > defaults work fine most of the time, so just use them. Turn the enable syscl > on just before you begin a run, and turn it off immediately afterwards. Make > sure to reset between reruns (rebooting to a new kernel is fine too!). > in ftp://ftp.cs.huji.ac.il/users/danny/lock.prof there 3 files: 7.1-100 host connected at 100 running -prerelease 7.1-1000same but connected at 1000 7.0-1000-stable with your 'patch' at 100 my benchmark didn't suffer from the profiling, average was about 9. at 1000 the benchmark got realy hit, average was around 12 for the patched, and 4 for the unpatched (less than at 100). danny ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
On Fri, 3 Oct 2008, Danny Braniss wrote: gladly, but have no idea how to do LOCK_PROFILING, so some pointers would be helpfull. The LOCK_PROFILING(9) man page isn't a bad starting point -- I find that the defaults work fine most of the time, so just use them. Turn the enable syscl on just before you begin a run, and turn it off immediately afterwards. Make sure to reset between reruns (rebooting to a new kernel is fine too!). as a side note, many years ago I checked out NFS/TCP and it was really bad, I even remember NetApp telling us to drop TCP, but now, things look rather better. Wonder what caused it. Well, the virtues of TCP become more apparent with higher network speeds, as the logic to fill pipes using TCP, manage flow control, etc, is a lot more sophisticated than what's in the RPC code for using UDP. The downsides to UDP are also becoming more apparent: as network speeds go up, fragmented UDP risks IP ID collisions which could lead to data corruption, or at the very least, dropped packets. We have changed the default for NFSv3 mounts to TCP in 8.x, and talked about doing it for 7.1; unfortunately the timing wasn't quite right, so it most likely will appear in 7.2. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
forget it about LOCK_PROFILING, I'm RTFM now :-) though some hints on values might be helpful. have a nice weekend, danny ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
>
> On Fri, 3 Oct 2008, Danny Braniss wrote:
>
> >> OK, so it looks like this was almost certainly the rwlock change. What
> >> happens if you pretty much universally substitute the following in
> >> udp_usrreq.c:
> >>
> >> Currently Change to
> >> - -
> >> INP_RLOCK INP_WLOCK
> >> INP_RUNLOCKINP_WUNLOCK
> >> INP_RLOCK_ASSERT INP_WLOCK_ASSERT
> >
> > I guess you were almost certainly correct :-) I did the global subst. on
> > the
> > udp_usrreq.c from 19/08, __FBSDID("$FreeBSD: src/sys/netinet/udp_usrreq.c,v
> > 1.218.2.3 2008/08/18 23:00:41 bz Exp $"); and now udp is fine again!
>
> OK. This is a change I'd rather not back out since it significantly improves
> performance for many other UDP workloads, so we need to figure out why it's
> hurting us so much here so that we know if there are reasonable alternatives.
>
> Would it be possible for you to do a run of the workload with both kernels
> using LOCK_PROFILING around the benchmark, and then we can compare lock
> contention in the two cases? What we often find is that relieving contention
> at one point causes new contention at another point, and if the primitive
> used
> at that point handles contention less well for whatever reason, performance
> can be reduced rather than improved. So maybe we're looking at an issue in
> the dispatched UDP code from so_upcall? Another less satisfying (and
> fundamentally more difficult) answer might be "something to do with the
> scheduler", but a bit more analysis may shed some light.
gladly, but have no idea how to do LOCK_PROFILING, so some pointers would be
helpfull.
as a side note, many years ago I checked out NFS/TCP and it was really bad,
I even remember NetApp telling us to drop TCP, but now, things look rather
better. Wonder what caused it.
danny
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
On Fri, 3 Oct 2008, Danny Braniss wrote:
OK, so it looks like this was almost certainly the rwlock change. What
happens if you pretty much universally substitute the following in
udp_usrreq.c:
Currently Change to
- -
INP_RLOCK INP_WLOCK
INP_RUNLOCK INP_WUNLOCK
INP_RLOCK_ASSERTINP_WLOCK_ASSERT
I guess you were almost certainly correct :-) I did the global subst. on the
udp_usrreq.c from 19/08, __FBSDID("$FreeBSD: src/sys/netinet/udp_usrreq.c,v
1.218.2.3 2008/08/18 23:00:41 bz Exp $"); and now udp is fine again!
OK. This is a change I'd rather not back out since it significantly improves
performance for many other UDP workloads, so we need to figure out why it's
hurting us so much here so that we know if there are reasonable alternatives.
Would it be possible for you to do a run of the workload with both kernels
using LOCK_PROFILING around the benchmark, and then we can compare lock
contention in the two cases? What we often find is that relieving contention
at one point causes new contention at another point, and if the primitive used
at that point handles contention less well for whatever reason, performance
can be reduced rather than improved. So maybe we're looking at an issue in
the dispatched UDP code from so_upcall? Another less satisfying (and
fundamentally more difficult) answer might be "something to do with the
scheduler", but a bit more analysis may shed some light.
Robert N M Watson
Computer Laboratory
University of Cambridge
danny
Robert N M Watson
Computer Laboratory
University of Cambridge
server is a NetApp:
kernel from 18/08/08 00:00:0 :
/- UDP // TCP ---/
1*512 38528 0.19s 83.50MB 0.20s 80.82MB/s
2*512 19264 0.21s 76.83MB 0.21s 77.57MB/s
4*512 9632 0.19s 85.51MB 0.22s 73.13MB/s
8*512 4816 0.19s 83.76MB 0.21s 75.84MB/s
16*512 2408 0.19s 83.99MB 0.21s 77.18MB/s
32*512 1204 0.19s 84.45MB 0.22s 71.79MB/s
64*512602 0.20s 79.98MB 0.20s 78.44MB/s
128*512301 0.18s 86.51MB 0.22s 71.53MB/s
256*512150 0.19s 82.83MB 0.20s 78.86MB/s
512*512 75 0.19s 82.77MB 0.21s 76.39MB/s
1024*512 37 0.19s 85.62MB 0.21s 76.64MB/s
2048*512 18 0.21s 77.72MB 0.20s 80.30MB/s
4096*512 9 0.26s 61.06MB 0.30s 53.79MB/s
8192*512 4 0.83s 19.20MB 0.41s 39.12MB/s
16384*512 2 0.84s 19.01MB 0.41s 39.03MB/s
32768*512 1 0.82s 19.59MB 0.39s 40.89MB/s
kernel from 19/08/08 00:00:00:
1*512 38528 0.45s 35.59MB 0.20s 81.43MB/s
2*512 19264 0.45s 35.56MB 0.20s 79.24MB/s
4*512 9632 0.49s 32.66MB 0.22s 73.72MB/s
8*512 4816 0.47s 34.06MB 0.21s 75.52MB/s
16*512 2408 0.53s 30.16MB 0.22s 72.58MB/s
32*512 1204 0.31s 51.68MB 0.40s 40.14MB/s
64*512602 0.43s 37.23MB 0.25s 63.57MB/s
128*512301 0.51s 31.39MB 0.26s 62.70MB/s
256*512150 0.47s 34.02MB 0.23s 69.06MB/s
512*512 75 0.47s 34.01MB 0.23s 70.52MB/s
1024*512 37 0.53s 30.12MB 0.22s 73.01MB/s
2048*512 18 0.55s 29.07MB 0.23s 70.64MB/s
4096*512 9 0.46s 34.69MB 0.21s 75.92MB/s
8192*512 4 0.81s 19.66MB 0.43s 36.89MB/s
16384*512 2 0.80s 19.99MB 0.40s 40.29MB/s
32768*512 1 1.11s 14.41MB 0.38s 42.56MB/s
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
>
> On Fri, 3 Oct 2008, Danny Braniss wrote:
>
> >>> it more difficult than I expected.
> >>> for one, the kernel date was missleading, the actual source update is the
> >>> key, so
> >>> the window of changes is now 28/July to 19/August. I have the diffs, but
> >>> nothing
> >>> yet seems relevant.
> >>>
> >>> on the other hand, I tried NFS/TCP, and there things seem ok, ie the
> >>> 'good' and the 'bad' give the same throughput, which seem to point to UDP
> >>> changes ...
> >>
> >> Can you post the network-numbers?
> > so I ran some more test, these are for writes IO:
>
> OK, so it looks like this was almost certainly the rwlock change. What
> happens if you pretty much universally substitute the following in
> udp_usrreq.c:
>
> Currently Change to
> - -
> INP_RLOCK INP_WLOCK
> INP_RUNLOCK INP_WUNLOCK
> INP_RLOCK_ASSERT INP_WLOCK_ASSERT
>
I guess you were almost certainly correct :-)
I did the global subst. on the udp_usrreq.c from 19/08,
__FBSDID("$FreeBSD: src/sys/netinet/udp_usrreq.c,v 1.218.2.3 2008/08/18
23:00:41 bz Exp $");
and now udp is fine again!
danny
> Robert N M Watson
> Computer Laboratory
> University of Cambridge
>
> >
> > server is a NetApp:
> >
> > kernel from 18/08/08 00:00:0 :
> >/- UDP // TCP ---/
> > 1*512 38528 0.19s 83.50MB 0.20s 80.82MB/s
> > 2*512 19264 0.21s 76.83MB 0.21s 77.57MB/s
> > 4*512 9632 0.19s 85.51MB 0.22s 73.13MB/s
> > 8*512 4816 0.19s 83.76MB 0.21s 75.84MB/s
> > 16*512 2408 0.19s 83.99MB 0.21s 77.18MB/s
> > 32*512 1204 0.19s 84.45MB 0.22s 71.79MB/s
> > 64*512602 0.20s 79.98MB 0.20s 78.44MB/s
> > 128*512301 0.18s 86.51MB 0.22s 71.53MB/s
> > 256*512150 0.19s 82.83MB 0.20s 78.86MB/s
> > 512*512 75 0.19s 82.77MB 0.21s 76.39MB/s
> >1024*512 37 0.19s 85.62MB 0.21s 76.64MB/s
> >2048*512 18 0.21s 77.72MB 0.20s 80.30MB/s
> >4096*512 9 0.26s 61.06MB 0.30s 53.79MB/s
> >8192*512 4 0.83s 19.20MB 0.41s 39.12MB/s
> > 16384*512 2 0.84s 19.01MB 0.41s 39.03MB/s
> > 32768*512 1 0.82s 19.59MB 0.39s 40.89MB/s
> >
> > kernel from 19/08/08 00:00:00:
> > 1*512 38528 0.45s 35.59MB 0.20s 81.43MB/s
> > 2*512 19264 0.45s 35.56MB 0.20s 79.24MB/s
> > 4*512 9632 0.49s 32.66MB 0.22s 73.72MB/s
> > 8*512 4816 0.47s 34.06MB 0.21s 75.52MB/s
> > 16*512 2408 0.53s 30.16MB 0.22s 72.58MB/s
> > 32*512 1204 0.31s 51.68MB 0.40s 40.14MB/s
> > 64*512602 0.43s 37.23MB 0.25s 63.57MB/s
> > 128*512301 0.51s 31.39MB 0.26s 62.70MB/s
> > 256*512150 0.47s 34.02MB 0.23s 69.06MB/s
> > 512*512 75 0.47s 34.01MB 0.23s 70.52MB/s
> >1024*512 37 0.53s 30.12MB 0.22s 73.01MB/s
> >2048*512 18 0.55s 29.07MB 0.23s 70.64MB/s
> >4096*512 9 0.46s 34.69MB 0.21s 75.92MB/s
> >8192*512 4 0.81s 19.66MB 0.43s 36.89MB/s
> > 16384*512 2 0.80s 19.99MB 0.40s 40.29MB/s
> > 32768*512 1 1.11s 14.41MB 0.38s 42.56MB/s
> >
> >
> >
> >
> >
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
On Fri, 3 Oct 2008, Danny Braniss wrote: it more difficult than I expected. for one, the kernel date was missleading, the actual source update is the key, so the window of changes is now 28/July to 19/August. I have the diffs, but nothing yet seems relevant. on the other hand, I tried NFS/TCP, and there things seem ok, ie the 'good' and the 'bad' give the same throughput, which seem to point to UDP changes ... Can you post the network-numbers? so I ran some more test, these are for writes IO: OK, so it looks like this was almost certainly the rwlock change. What happens if you pretty much universally substitute the following in udp_usrreq.c: Currently Change to - - INP_RLOCK INP_WLOCK INP_RUNLOCK INP_WUNLOCK INP_RLOCK_ASSERTINP_WLOCK_ASSERT Robert N M Watson Computer Laboratory University of Cambridge server is a NetApp: kernel from 18/08/08 00:00:0 : /- UDP // TCP ---/ 1*512 38528 0.19s 83.50MB 0.20s 80.82MB/s 2*512 19264 0.21s 76.83MB 0.21s 77.57MB/s 4*512 9632 0.19s 85.51MB 0.22s 73.13MB/s 8*512 4816 0.19s 83.76MB 0.21s 75.84MB/s 16*512 2408 0.19s 83.99MB 0.21s 77.18MB/s 32*512 1204 0.19s 84.45MB 0.22s 71.79MB/s 64*512602 0.20s 79.98MB 0.20s 78.44MB/s 128*512301 0.18s 86.51MB 0.22s 71.53MB/s 256*512150 0.19s 82.83MB 0.20s 78.86MB/s 512*512 75 0.19s 82.77MB 0.21s 76.39MB/s 1024*512 37 0.19s 85.62MB 0.21s 76.64MB/s 2048*512 18 0.21s 77.72MB 0.20s 80.30MB/s 4096*512 9 0.26s 61.06MB 0.30s 53.79MB/s 8192*512 4 0.83s 19.20MB 0.41s 39.12MB/s 16384*512 2 0.84s 19.01MB 0.41s 39.03MB/s 32768*512 1 0.82s 19.59MB 0.39s 40.89MB/s kernel from 19/08/08 00:00:00: 1*512 38528 0.45s 35.59MB 0.20s 81.43MB/s 2*512 19264 0.45s 35.56MB 0.20s 79.24MB/s 4*512 9632 0.49s 32.66MB 0.22s 73.72MB/s 8*512 4816 0.47s 34.06MB 0.21s 75.52MB/s 16*512 2408 0.53s 30.16MB 0.22s 72.58MB/s 32*512 1204 0.31s 51.68MB 0.40s 40.14MB/s 64*512602 0.43s 37.23MB 0.25s 63.57MB/s 128*512301 0.51s 31.39MB 0.26s 62.70MB/s 256*512150 0.47s 34.02MB 0.23s 69.06MB/s 512*512 75 0.47s 34.01MB 0.23s 70.52MB/s 1024*512 37 0.53s 30.12MB 0.22s 73.01MB/s 2048*512 18 0.55s 29.07MB 0.23s 70.64MB/s 4096*512 9 0.46s 34.69MB 0.21s 75.92MB/s 8192*512 4 0.81s 19.66MB 0.43s 36.89MB/s 16384*512 2 0.80s 19.99MB 0.40s 40.29MB/s 32768*512 1 1.11s 14.41MB 0.38s 42.56MB/s ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
> > it more difficult than I expected. > > for one, the kernel date was missleading, the actual source update is the > > key, so > > the window of changes is now 28/July to 19/August. I have the diffs, but > > nothing > > yet seems relevant. > > > > on the other hand, I tried NFS/TCP, and there things seem ok, ie the 'good' > > and the 'bad' > > give the same throughput, which seem to point to UDP changes ... > > Can you post the network-numbers? so I ran some more test, these are for writes IO: server is a NetApp: kernel from 18/08/08 00:00:0 : /- UDP // TCP ---/ 1*512 38528 0.19s 83.50MB 0.20s 80.82MB/s 2*512 19264 0.21s 76.83MB 0.21s 77.57MB/s 4*512 9632 0.19s 85.51MB 0.22s 73.13MB/s 8*512 4816 0.19s 83.76MB 0.21s 75.84MB/s 16*512 2408 0.19s 83.99MB 0.21s 77.18MB/s 32*512 1204 0.19s 84.45MB 0.22s 71.79MB/s 64*512602 0.20s 79.98MB 0.20s 78.44MB/s 128*512301 0.18s 86.51MB 0.22s 71.53MB/s 256*512150 0.19s 82.83MB 0.20s 78.86MB/s 512*512 75 0.19s 82.77MB 0.21s 76.39MB/s 1024*512 37 0.19s 85.62MB 0.21s 76.64MB/s 2048*512 18 0.21s 77.72MB 0.20s 80.30MB/s 4096*512 9 0.26s 61.06MB 0.30s 53.79MB/s 8192*512 4 0.83s 19.20MB 0.41s 39.12MB/s 16384*512 2 0.84s 19.01MB 0.41s 39.03MB/s 32768*512 1 0.82s 19.59MB 0.39s 40.89MB/s kernel from 19/08/08 00:00:00: 1*512 38528 0.45s 35.59MB 0.20s 81.43MB/s 2*512 19264 0.45s 35.56MB 0.20s 79.24MB/s 4*512 9632 0.49s 32.66MB 0.22s 73.72MB/s 8*512 4816 0.47s 34.06MB 0.21s 75.52MB/s 16*512 2408 0.53s 30.16MB 0.22s 72.58MB/s 32*512 1204 0.31s 51.68MB 0.40s 40.14MB/s 64*512602 0.43s 37.23MB 0.25s 63.57MB/s 128*512301 0.51s 31.39MB 0.26s 62.70MB/s 256*512150 0.47s 34.02MB 0.23s 69.06MB/s 512*512 75 0.47s 34.01MB 0.23s 70.52MB/s 1024*512 37 0.53s 30.12MB 0.22s 73.01MB/s 2048*512 18 0.55s 29.07MB 0.23s 70.64MB/s 4096*512 9 0.46s 34.69MB 0.21s 75.92MB/s 8192*512 4 0.81s 19.66MB 0.43s 36.89MB/s 16384*512 2 0.80s 19.99MB 0.40s 40.29MB/s 32768*512 1 1.11s 14.41MB 0.38s 42.56MB/s ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
On Mon, 29 Sep 2008, Oliver Fromme wrote: Danny Braniss wrote: > Grr, there goes binary search theory out of the window, > So far I have managed to pinpoint the day that the changes affect the > throughput: > 18/08/08 00:00:00 19/08/08 00:00:00 > (I assume cvs's date is GMT). > now would be a good time for some help, specially how to undo changes, my > knowledge of csup/cvs are close to zero. So you've nailed to down to this 24-hour window: http://www.secnetix.de/olli/FreeBSD/svnews/?day=2008-08-18&p=/stable/7 I'm afraid that r181822 by rwatson is the most likely candidate that might be causing the regression. If we can confirm that it was that specific change, then I can create a patch to restore exclusive locking for UDP and we can see if it was the general move to rwlocking, or specifically the read-locking of certain data structures. Perhaps what we've done is moved contention from a mutex to a sleep lock, reducing the efficiency of handling contention? Adding Kris to the CC line because he often has useful insights on this sort of thing. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
Danny Braniss wrote: > Grr, there goes binary search theory out of the window, > So far I have managed to pinpoint the day that the changes affect the > throughput: > 18/08/08 00:00:00 19/08/08 00:00:00 > (I assume cvs's date is GMT). > now would be a good time for some help, specially how to undo changes, my > knowledge of csup/cvs are close to zero. So you've nailed to down to this 24-hour window: http://www.secnetix.de/olli/FreeBSD/svnews/?day=2008-08-18&p=/stable/7 I'm afraid that r181822 by rwatson is the most likely candidate that might be causing the regression. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "C is quirky, flawed, and an enormous success." -- Dennis M. Ritchie. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
> it more difficult than I expected. > for one, the kernel date was missleading, the actual source update is the > key, so > the window of changes is now 28/July to 19/August. I have the diffs, but > nothing > yet seems relevant. > > on the other hand, I tried NFS/TCP, and there things seem ok, ie the 'good' > and the 'bad' > give the same throughput, which seem to point to UDP changes ... Can you post the network-numbers? -- regards Claus When lenity and cruelty play for a kingdom, the gentler gamester is the soonest winner. Shakespeare ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
> > it more difficult than I expected. > > for one, the kernel date was missleading, the actual source update is the > > key, so > > the window of changes is now 28/July to 19/August. I have the diffs, but > > nothing > > yet seems relevant. > > > > on the other hand, I tried NFS/TCP, and there things seem ok, ie the 'good' > > and the 'bad' > > give the same throughput, which seem to point to UDP changes ... > > Can you post the network-numbers? [again :-] > > Writing 16 MB file > > BSCount / 7.0 --/ / 7.1 -/ should now read: / Aug 18 --/ /--- Aug 19 / > >1*512 32768 0.16s 98.11MB/s 0.43s 37.18MB/s > >2*512 16384 0.17s 92.04MB/s 0.46s 34.79MB/s > >4*512 8192 0.16s 101.88MB/s 0.43s 37.26MB/s > >8*512 4096 0.16s 99.86MB/s 0.44s 36.41MB/s > > 16*512 2048 0.16s 100.11MB/s 0.50s 32.03MB/s > > 32*512 1024 0.26s 61.71MB/s 0.46s 34.79MB/s > > 64*512512 0.22s 71.45MB/s 0.45s 35.41MB/s > > 128*512256 0.21s 77.84MB/s 0.51s 31.34MB/s > > 256*512128 0.19s 82.47MB/s 0.43s 37.22MB/s > > 512*512 64 0.18s 87.77MB/s 0.49s 32.69MB/s > > 1024*512 32 0.18s 89.24MB/s 0.47s 34.02MB/s > > 2048*512 16 0.17s 91.81MB/s 0.30s 53.41MB/s > > 4096*512 8 0.16s 100.56MB/s 0.42s 38.07MB/s > > 8192*512 4 0.82s 19.56MB/s 0.80s 19.95MB/s > >16384*512 2 0.82s 19.63MB/s 0.95s 16.80MB/s > >32768*512 1 0.81s 19.69MB/s 0.96s 16.64MB/s > > > > Average: 75.8633.00 ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
> > On Fri, 26 Sep 2008, Danny Braniss wrote: > > > > > after more testing, it seems it's related to changes made between Aug 4 > > > and > > > Aug 29 ie, a kernel built on Aug 4 works fine, Aug 29 is slow. I'l now > > > try > > > and close the gap. > > > > I think this is the best way forward -- skimming August changes, there are > > a > > number of candidate commits, including retuning of UDP hashes by mav, my > > rwlock changes, changes to mbuf chain handling, etc. > > it more difficult than I expected. > for one, the kernel date was missleading, the actual source update is the > key, so > the window of changes is now 28/July to 19/August. I have the diffs, but > nothing > yet seems relevant. > > on the other hand, I tried NFS/TCP, and there things seem ok, ie the 'good' > and the 'bad' > give the same throughput, which seem to point to UDP changes ... > > danny Grr, there goes binary search theory out of the window, So far I have managed to pinpoint the day that the changes affect the throughput: 18/08/08 00:00:00 19/08/08 00:00:00 (I assume cvs's date is GMT). now would be a good time for some help, specially how to undo changes, my knowledge of csup/cvs are close to zero. danny ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
Danny Braniss wrote: I know, but I get about 1mgb, which seems somewhat low :-( If you don't tell iperf how much bandwidth to use for a UDP test, it defaults to 1Mbps. See -b option. http://dast.nlanr.net/projects/Iperf/iperfdocs_1.7.0.php#bandwidth --eli -- Eli Dart ESnet Network Engineering Group Lawrence Berkeley National Laboratory PGP Key fingerprint = C970 F8D3 CFDD 8FFF 5486 343A 2D31 4478 5F82 B2B3 ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
> On Fri, 26 Sep 2008, Danny Braniss wrote: > > > after more testing, it seems it's related to changes made between Aug 4 and > > Aug 29 ie, a kernel built on Aug 4 works fine, Aug 29 is slow. I'l now try > > and close the gap. > > I think this is the best way forward -- skimming August changes, there are a > number of candidate commits, including retuning of UDP hashes by mav, my > rwlock changes, changes to mbuf chain handling, etc. it more difficult than I expected. for one, the kernel date was missleading, the actual source update is the key, so the window of changes is now 28/July to 19/August. I have the diffs, but nothing yet seems relevant. on the other hand, I tried NFS/TCP, and there things seem ok, ie the 'good' and the 'bad' give the same throughput, which seem to point to UDP changes ... danny ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
On Fri, 26 Sep 2008, Danny Braniss wrote: after more testing, it seems it's related to changes made between Aug 4 and Aug 29 ie, a kernel built on Aug 4 works fine, Aug 29 is slow. I'l now try and close the gap. I think this is the best way forward -- skimming August changes, there are a number of candidate commits, including retuning of UDP hashes by mav, my rwlock changes, changes to mbuf chain handling, etc. Thanks, Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
> --==_Exmh_1222467420_5817P > Content-Type: text/plain; charset=us-ascii > Content-Disposition: inline > > David, > > You beat me to it. > > Danny, read the iperf man page: >-b, --bandwidth n[KM] > set target bandwidth to n bits/sec (default 1 Mbit/sec). This > setting requires UDP (-u). > > The page needs updating, though. It should read "-b, --bandwidth > n[KMG]. It also does NOT require -u. If you use -b, UDP is assumed. I did RTFM(*), but when i tried it just wouldn't work, I tried today and it's actually working - so don't RTFM before coffee! btw, even though iperf sucks, netperf udp tends to bring the server down to it's knees. danny PS: * - i don't seem to have the iperf man, all I have is iperf -h ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
:how can I see the IP fragment reassembly statistics? : :thanks, : danny netstat -s Also look for unexpected dropped packets, dropped fragments, and errors during the test and such, they are counted in the statistics as well. -Matt Matthew Dillon <[EMAIL PROTECTED]> ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
> > :>-vfs.nfs.realign_test: 22141777 > :>+vfs.nfs.realign_test: 498351 > :> > :>-vfs.nfsrv.realign_test: 5005908 > :>+vfs.nfsrv.realign_test: 0 > :> > :>+vfs.nfsrv.commit_miss: 0 > :>+vfs.nfsrv.commit_blks: 0 > :> > :> changing them did nothing - or at least with respect to nfs throughput :-) > : > :I'm not sure what any of these do, as NFS is a bit out of my league. > ::-) I'll be following this thread though! > : > :-- > :| Jeremy Chadwickjdc at parodius.com | > > A non-zero nfs_realign_count is bad, it means NFS had to copy the > mbuf chain to fix the alignment. nfs_realign_test is just the > number of times it checked. So nfs_realign_test is irrelevant. > it's nfs_realign_count that matters. > it's zero, so I guess I'm ok there. funny though, on my 'good' machine, vfs.nfsrv.realign_test: 5862999 and on the slow one, it's 0 - but then again the good one has been up for several days. > Several things can cause NFS payloads to be improperly aligned. > Anything from older network drivers which can't start DMA on a > 2-byte boundary, resulting in the 14-byte encapsulation header > causing improper alignment of the IP header & payload, to rpc > embedded in NFS TCP streams winding up being misaligned. > > Modern network hardware either support 2-byte-aligned DMA, allowing > the encapsulation to be 2-byte aligned so the payload winds up being > 4-byte aligned, or support DMA chaining allowing the payload to be > placed in its own mbuf, or pad, etc. > > -- > > One thing I would check is to be sure a couple of nfsiod's are running > on the client when doing your tests. If none are running the RPCs wind > up being more synchronous and less pipelined. Another thing I would > check is IP fragment reassembly statistics (for UDP) - there should be > none for TCP connections no matter what the NFS I/O size selected. > ahh, nfsiod, it seems that it's now dynamicaly started! at least none show when host is idle, after i run my tests there are 20! with ppid 0 need to refresh my NFS knowledge. how can I see the IP fragment reassembly statistics? > (It does seem more likely to be scheduler-related, though). > tend to agree, I tried bith ULE/BSD, but the badness is there. > -Matt > thanks, danny ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
David, You beat me to it. Danny, read the iperf man page: -b, --bandwidth n[KM] set target bandwidth to n bits/sec (default 1 Mbit/sec). This setting requires UDP (-u). The page needs updating, though. It should read "-b, --bandwidth n[KMG]. It also does NOT require -u. If you use -b, UDP is assumed. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: [EMAIL PROTECTED] Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751 pgpOo3ttHcEYe.pgp Description: PGP signature
Re: bad NFS/UDP performance
On Fri, Sep 26, 2008 at 04:35:17PM +0300, Danny Braniss wrote: > I know, but I get about 1mgb, which seems somewhat low :-( Since UDP has no way to know how fast to send, you need to tell iperf how fast to send the packets. I think 1Mbps is the default speed. David. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
:> -vfs.nfs.realign_test: 22141777 :> +vfs.nfs.realign_test: 498351 :> :> -vfs.nfsrv.realign_test: 5005908 :> +vfs.nfsrv.realign_test: 0 :> :> +vfs.nfsrv.commit_miss: 0 :> +vfs.nfsrv.commit_blks: 0 :> :> changing them did nothing - or at least with respect to nfs throughput :-) : :I'm not sure what any of these do, as NFS is a bit out of my league. ::-) I'll be following this thread though! : :-- :| Jeremy Chadwickjdc at parodius.com | A non-zero nfs_realign_count is bad, it means NFS had to copy the mbuf chain to fix the alignment. nfs_realign_test is just the number of times it checked. So nfs_realign_test is irrelevant. it's nfs_realign_count that matters. Several things can cause NFS payloads to be improperly aligned. Anything from older network drivers which can't start DMA on a 2-byte boundary, resulting in the 14-byte encapsulation header causing improper alignment of the IP header & payload, to rpc embedded in NFS TCP streams winding up being misaligned. Modern network hardware either support 2-byte-aligned DMA, allowing the encapsulation to be 2-byte aligned so the payload winds up being 4-byte aligned, or support DMA chaining allowing the payload to be placed in its own mbuf, or pad, etc. -- One thing I would check is to be sure a couple of nfsiod's are running on the client when doing your tests. If none are running the RPCs wind up being more synchronous and less pipelined. Another thing I would check is IP fragment reassembly statistics (for UDP) - there should be none for TCP connections no matter what the NFS I/O size selected. (It does seem more likely to be scheduler-related, though). -Matt ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
> On Fri, Sep 26, 2008 at 12:27:08PM +0300, Danny Braniss wrote: > > > On Fri, Sep 26, 2008 at 10:04:16AM +0300, Danny Braniss wrote: > > > > Hi, > > > > There seems to be some serious degradation in performance. > > > > Under 7.0 I get about 90 MB/s (on write), while, on the same machine > > > > under 7.1 it drops to 20! > > > > Any ideas? > > > > > > 1) Network card driver changes, > > could be, but at least iperf/tcp is ok - can't get udp numbers, do you > > know of any tool to measure udp performance? > > BTW, I also checked on different hardware, and the badness is there. > > According to INDEX, benchmarks/iperf does UDP bandwidth testing. > > benchmarks/nttcp should as well. > > What network card is in use? If Intel, what driver version (should be > in dmesg). > > > > 2) This could be relevant, but rwatson@ will need to help determine > > >that. > > > > > > http://lists.freebsd.org/pipermail/freebsd-stable/2008-September/045109.html > > > > gut feeling is that it's somewhere else: > > > > Writing 16 MB file > > BSCount / 7.0 --/ / 7.1 -/ > >1*512 32768 0.16s 98.11MB/s 0.43s 37.18MB/s > >2*512 16384 0.17s 92.04MB/s 0.46s 34.79MB/s > >4*512 8192 0.16s 101.88MB/s 0.43s 37.26MB/s > >8*512 4096 0.16s 99.86MB/s 0.44s 36.41MB/s > > 16*512 2048 0.16s 100.11MB/s 0.50s 32.03MB/s > > 32*512 1024 0.26s 61.71MB/s 0.46s 34.79MB/s > > 64*512512 0.22s 71.45MB/s 0.45s 35.41MB/s > > 128*512256 0.21s 77.84MB/s 0.51s 31.34MB/s > > 256*512128 0.19s 82.47MB/s 0.43s 37.22MB/s > > 512*512 64 0.18s 87.77MB/s 0.49s 32.69MB/s > > 1024*512 32 0.18s 89.24MB/s 0.47s 34.02MB/s > > 2048*512 16 0.17s 91.81MB/s 0.30s 53.41MB/s > > 4096*512 8 0.16s 100.56MB/s 0.42s 38.07MB/s > > 8192*512 4 0.82s 19.56MB/s 0.80s 19.95MB/s > >16384*512 2 0.82s 19.63MB/s 0.95s 16.80MB/s > >32768*512 1 0.81s 19.69MB/s 0.96s 16.64MB/s > > > > Average: 75.8633.00 > > > > the nfs filer is a NetWork Appliance, and is in use, so i get fluctuations > > in > > the > > measurements, but the relation are similar, good on 7.0, bad on 7.1 > > Do you have any NFS-related tunings in /etc/rc.conf or /etc/sysctl.conf? > after more testing, it seems it's related to changes made between Aug 4 and Aug 29 ie, a kernel built on Aug 4 works fine, Aug 29 is slow. I'l now try and close the gap. danny ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
On Fri, Sep 26, 2008 at 04:35:17PM +0300, Danny Braniss wrote: > > On Fri, Sep 26, 2008 at 12:27:08PM +0300, Danny Braniss wrote: > > > > On Fri, Sep 26, 2008 at 10:04:16AM +0300, Danny Braniss wrote: > > > > > Hi, > > > > > There seems to be some serious degradation in performance. > > > > > Under 7.0 I get about 90 MB/s (on write), while, on the same machine > > > > > under 7.1 it drops to 20! > > > > > Any ideas? > > > > > > > > 1) Network card driver changes, > > > could be, but at least iperf/tcp is ok - can't get udp numbers, do you > > > know of any tool to measure udp performance? > > > BTW, I also checked on different hardware, and the badness is there. > > > > According to INDEX, benchmarks/iperf does UDP bandwidth testing. > > I know, but I get about 1mgb, which seems somewhat low :-( > > > > > benchmarks/nttcp should as well. > > > > What network card is in use? If Intel, what driver version (should be > > in dmesg). > > bge: > and > bce: > and intels, but haven't tested there yet. Both bge(4) and bce(4) claim to support checksum offloading. You might try disabling it (ifconfig ... -txcsum -rxcsum) to see if things improve. If not, more troubleshooting is needed. You might also try turning off TSO if it's supported (check your ifconfig output for TSO in the options=<> section. Then use ifconfig ... -tso) > > Do you have any NFS-related tunings in /etc/rc.conf or /etc/sysctl.conf? > > > no, but diffing the sysctl show: > > -vfs.nfs.realign_test: 22141777 > +vfs.nfs.realign_test: 498351 > > -vfs.nfsrv.realign_test: 5005908 > +vfs.nfsrv.realign_test: 0 > > +vfs.nfsrv.commit_miss: 0 > +vfs.nfsrv.commit_blks: 0 > > changing them did nothing - or at least with respect to nfs throughput :-) I'm not sure what any of these do, as NFS is a bit out of my league. :-) I'll be following this thread though! -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
On Friday 26 September 2008 03:04:16 am Danny Braniss wrote: > Hi, > There seems to be some serious degradation in performance. > Under 7.0 I get about 90 MB/s (on write), while, on the same machine > under 7.1 it drops to 20! > Any ideas? > > thanks, > danny Perhaps use nfsstat to see if 7.1 is performing more on-the-wire requests? Also, if you can, do a binary search to narrow down when the regression occurred in RELENG_7. -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
> On Fri, 2008-09-26 at 10:04 +0300, Danny Braniss wrote: > > Hi, > > There seems to be some serious degradation in performance. > > Under 7.0 I get about 90 MB/s (on write), while, on the same machine > > under 7.1 it drops to 20! > > Any ideas? > > The scheduler has been changed to ULE, and NFS has historically been > very sensitive to changes like that. You could try switching back to > the 4BSD scheduler and seeing if that makes a difference. If it does, > toggling PREEMPTION would also be interesting to see the results of. > > Gavin I'm testing 7.0-stable vs 7.1-prerelease, and both have ULE. BTW, the nfs client hosts I'm testing are idle. danny ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
> On Fri, Sep 26, 2008 at 12:27:08PM +0300, Danny Braniss wrote: > > > On Fri, Sep 26, 2008 at 10:04:16AM +0300, Danny Braniss wrote: > > > > Hi, > > > > There seems to be some serious degradation in performance. > > > > Under 7.0 I get about 90 MB/s (on write), while, on the same machine > > > > under 7.1 it drops to 20! > > > > Any ideas? > > > > > > 1) Network card driver changes, > > could be, but at least iperf/tcp is ok - can't get udp numbers, do you > > know of any tool to measure udp performance? > > BTW, I also checked on different hardware, and the badness is there. > > According to INDEX, benchmarks/iperf does UDP bandwidth testing. I know, but I get about 1mgb, which seems somewhat low :-( > > benchmarks/nttcp should as well. > > What network card is in use? If Intel, what driver version (should be > in dmesg). bge: and bce: and intels, but haven't tested there yet. > > > > 2) This could be relevant, but rwatson@ will need to help determine > > >that. > > > > > > http://lists.freebsd.org/pipermail/freebsd-stable/2008-September/045109.html > > > > gut feeling is that it's somewhere else: > > > > Writing 16 MB file > > BSCount / 7.0 --/ / 7.1 -/ > >1*512 32768 0.16s 98.11MB/s 0.43s 37.18MB/s > >2*512 16384 0.17s 92.04MB/s 0.46s 34.79MB/s > >4*512 8192 0.16s 101.88MB/s 0.43s 37.26MB/s > >8*512 4096 0.16s 99.86MB/s 0.44s 36.41MB/s > > 16*512 2048 0.16s 100.11MB/s 0.50s 32.03MB/s > > 32*512 1024 0.26s 61.71MB/s 0.46s 34.79MB/s > > 64*512512 0.22s 71.45MB/s 0.45s 35.41MB/s > > 128*512256 0.21s 77.84MB/s 0.51s 31.34MB/s > > 256*512128 0.19s 82.47MB/s 0.43s 37.22MB/s > > 512*512 64 0.18s 87.77MB/s 0.49s 32.69MB/s > > 1024*512 32 0.18s 89.24MB/s 0.47s 34.02MB/s > > 2048*512 16 0.17s 91.81MB/s 0.30s 53.41MB/s > > 4096*512 8 0.16s 100.56MB/s 0.42s 38.07MB/s > > 8192*512 4 0.82s 19.56MB/s 0.80s 19.95MB/s > >16384*512 2 0.82s 19.63MB/s 0.95s 16.80MB/s > >32768*512 1 0.81s 19.69MB/s 0.96s 16.64MB/s > > > > Average: 75.8633.00 > > > > the nfs filer is a NetWork Appliance, and is in use, so i get fluctuations > > in > > the > > measurements, but the relation are similar, good on 7.0, bad on 7.1 > > Do you have any NFS-related tunings in /etc/rc.conf or /etc/sysctl.conf? > no, but diffing the sysctl show: -vfs.nfs.realign_test: 22141777 +vfs.nfs.realign_test: 498351 -vfs.nfsrv.realign_test: 5005908 +vfs.nfsrv.realign_test: 0 +vfs.nfsrv.commit_miss: 0 +vfs.nfsrv.commit_blks: 0 changing them did nothing - or at least with respect to nfs throughput :-) danny ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
On Fri, 2008-09-26 at 10:04 +0300, Danny Braniss wrote: > Hi, > There seems to be some serious degradation in performance. > Under 7.0 I get about 90 MB/s (on write), while, on the same machine > under 7.1 it drops to 20! > Any ideas? The scheduler has been changed to ULE, and NFS has historically been very sensitive to changes like that. You could try switching back to the 4BSD scheduler and seeing if that makes a difference. If it does, toggling PREEMPTION would also be interesting to see the results of. Gavin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
>There seems to be some serious degradation in performance. > Under 7.0 I get about 90 MB/s (on write), while, on the same machine > under 7.1 it drops to 20! > Any ideas? Can you compare performanc with tcp? -- regards Claus When lenity and cruelty play for a kingdom, the gentler gamester is the soonest winner. Shakespeare ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
On Fri, Sep 26, 2008 at 12:27:08PM +0300, Danny Braniss wrote: > > On Fri, Sep 26, 2008 at 10:04:16AM +0300, Danny Braniss wrote: > > > Hi, > > > There seems to be some serious degradation in performance. > > > Under 7.0 I get about 90 MB/s (on write), while, on the same machine > > > under 7.1 it drops to 20! > > > Any ideas? > > > > 1) Network card driver changes, > could be, but at least iperf/tcp is ok - can't get udp numbers, do you > know of any tool to measure udp performance? > BTW, I also checked on different hardware, and the badness is there. According to INDEX, benchmarks/iperf does UDP bandwidth testing. benchmarks/nttcp should as well. What network card is in use? If Intel, what driver version (should be in dmesg). > > 2) This could be relevant, but rwatson@ will need to help determine > >that. > > > > http://lists.freebsd.org/pipermail/freebsd-stable/2008-September/045109.html > > gut feeling is that it's somewhere else: > > Writing 16 MB file > BSCount / 7.0 --/ / 7.1 -/ >1*512 32768 0.16s 98.11MB/s 0.43s 37.18MB/s >2*512 16384 0.17s 92.04MB/s 0.46s 34.79MB/s >4*512 8192 0.16s 101.88MB/s 0.43s 37.26MB/s >8*512 4096 0.16s 99.86MB/s 0.44s 36.41MB/s > 16*512 2048 0.16s 100.11MB/s 0.50s 32.03MB/s > 32*512 1024 0.26s 61.71MB/s 0.46s 34.79MB/s > 64*512512 0.22s 71.45MB/s 0.45s 35.41MB/s > 128*512256 0.21s 77.84MB/s 0.51s 31.34MB/s > 256*512128 0.19s 82.47MB/s 0.43s 37.22MB/s > 512*512 64 0.18s 87.77MB/s 0.49s 32.69MB/s > 1024*512 32 0.18s 89.24MB/s 0.47s 34.02MB/s > 2048*512 16 0.17s 91.81MB/s 0.30s 53.41MB/s > 4096*512 8 0.16s 100.56MB/s 0.42s 38.07MB/s > 8192*512 4 0.82s 19.56MB/s 0.80s 19.95MB/s >16384*512 2 0.82s 19.63MB/s 0.95s 16.80MB/s >32768*512 1 0.81s 19.69MB/s 0.96s 16.64MB/s > > Average: 75.8633.00 > > the nfs filer is a NetWork Appliance, and is in use, so i get fluctuations in > the > measurements, but the relation are similar, good on 7.0, bad on 7.1 Do you have any NFS-related tunings in /etc/rc.conf or /etc/sysctl.conf? -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
> On Fri, Sep 26, 2008 at 10:04:16AM +0300, Danny Braniss wrote: > > Hi, > > There seems to be some serious degradation in performance. > > Under 7.0 I get about 90 MB/s (on write), while, on the same machine > > under 7.1 it drops to 20! > > Any ideas? > > 1) Network card driver changes, could be, but at least iperf/tcp is ok - can't get udp numbers, do you know of any tool to measure udp performance? BTW, I also checked on different hardware, and the badness is there. > > 2) This could be relevant, but rwatson@ will need to help determine >that. > > http://lists.freebsd.org/pipermail/freebsd-stable/2008-September/045109.html gut feeling is that it's somewhere else: Writing 16 MB file BSCount / 7.0 --/ / 7.1 -/ 1*512 32768 0.16s 98.11MB/s 0.43s 37.18MB/s 2*512 16384 0.17s 92.04MB/s 0.46s 34.79MB/s 4*512 8192 0.16s 101.88MB/s 0.43s 37.26MB/s 8*512 4096 0.16s 99.86MB/s 0.44s 36.41MB/s 16*512 2048 0.16s 100.11MB/s 0.50s 32.03MB/s 32*512 1024 0.26s 61.71MB/s 0.46s 34.79MB/s 64*512512 0.22s 71.45MB/s 0.45s 35.41MB/s 128*512256 0.21s 77.84MB/s 0.51s 31.34MB/s 256*512128 0.19s 82.47MB/s 0.43s 37.22MB/s 512*512 64 0.18s 87.77MB/s 0.49s 32.69MB/s 1024*512 32 0.18s 89.24MB/s 0.47s 34.02MB/s 2048*512 16 0.17s 91.81MB/s 0.30s 53.41MB/s 4096*512 8 0.16s 100.56MB/s 0.42s 38.07MB/s 8192*512 4 0.82s 19.56MB/s 0.80s 19.95MB/s 16384*512 2 0.82s 19.63MB/s 0.95s 16.80MB/s 32768*512 1 0.81s 19.69MB/s 0.96s 16.64MB/s Average: 75.8633.00 the nfs filer is a NetWork Appliance, and is in use, so i get fluctuations in the measurements, but the relation are similar, good on 7.0, bad on 7.1 Cheers, danny ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
On Fri, Sep 26, 2008 at 10:04:16AM +0300, Danny Braniss wrote: > Hi, > There seems to be some serious degradation in performance. > Under 7.0 I get about 90 MB/s (on write), while, on the same machine > under 7.1 it drops to 20! > Any ideas? 1) Network card driver changes, 2) This could be relevant, but rwatson@ will need to help determine that. http://lists.freebsd.org/pipermail/freebsd-stable/2008-September/045109.html -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
bad NFS/UDP performance
Hi, There seems to be some serious degradation in performance. Under 7.0 I get about 90 MB/s (on write), while, on the same machine under 7.1 it drops to 20! Any ideas? thanks, danny ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
