Re: Why is NFSv4 so slow?

2010-09-13 Thread Oliver Fromme
Rick Macklem wrote: Btw, if anyone who didn't see the posting on freebsd-fs and would like to run a quick test, it would be appreciated. Bascially do both kinds of mount using a FreeBSD8.1 or later client and then read a greater than 100Mbyte file with dd. # mount -t nfs -o nfsv3

Re: Why is NFSv4 so slow?

2010-09-13 Thread Rick Macklem
Ok ... NFS server: - FreeBSD 8.1-PRERELEASE-20100620 i386 - intel Atom 330 (1.6 GHz dual-core with HT -- 4-way SMP) - 4 GB RAM - re0: RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet NFS client: - FreeBSD 8.1-STABLE-20100908 i386 - AMD Phenom II X6 1055T (2.8 GHz + Turbo Core,

Re: Why is NFSv4 so slow?

2010-09-13 Thread Rick C. Petty
On Mon, Sep 13, 2010 at 11:15:34AM -0400, Rick Macklem wrote: instead of from the local cache. I also made sure that the file was in the cache on the server, so the server's disk speed is irrelevant. snip So, nfs is roughly twice as fast as newnfs, indeed. Hmm, I have the same

Re: Why is NFSv4 so slow?

2010-09-13 Thread Goran Lowkrantz
--On September 12, 2010 11:44:40 -0400 Rick Macklem rmack...@uoguelph.ca wrote: On Wed, Sep 01, 2010 at 11:46:30AM -0400, Rick Macklem wrote: snip My results seems to confirm a factor of two (or 1.5) but it's stable: new nfs nfsv4 369792 bytes transferred in 71.932692 secs

Re: Why is NFSv4 so slow?

2010-09-12 Thread Rick Macklem
On Wed, Sep 01, 2010 at 11:46:30AM -0400, Rick Macklem wrote: I am experiencing similar issues with newnfs: 1) I have two clients that each get around 0.5MiB/s to 2.6MiB/s reading from the NFS4-share on Gbit-Lan 2) Mounting with -t newnfs -o nfsv3 results in no performance

Re: Why is NFSv4 so slow?

2010-09-04 Thread Rick Macklem
Do you (or will you soon) have some patches I/we could test? I'm willing to try anything to avoid mounting ten or so subdirectories in each of my mount points. Attached is a small patch for the only difference I can spot in the read code between the regular and experimental NFS client. I

Re: Why is NFSv4 so slow?

2010-09-04 Thread Rick Macklem
- Original Message - On Wed, Sep 01, 2010 at 11:46:30AM -0400, Rick Macklem wrote: I am experiencing similar issues with newnfs: 1) I have two clients that each get around 0.5MiB/s to 2.6MiB/s reading from the NFS4-share on Gbit-Lan 2) Mounting with -t newnfs -o

Re: Why is NFSv4 so slow?

2010-09-03 Thread Rick C. Petty
On Mon, Aug 30, 2010 at 09:59:38PM -0400, Rick Macklem wrote: I don't tune anything with sysctl, I just use what I get from an install from CD onto i386 hardware. (I don't even bother to increase kern.ipc.maxsockbuf although I suggest that in the mount message.) Sure. But maybe you don't

Re: Why is NFSv4 so slow?

2010-09-03 Thread Rick C. Petty
On Wed, Sep 01, 2010 at 11:46:30AM -0400, Rick Macklem wrote: I am experiencing similar issues with newnfs: 1) I have two clients that each get around 0.5MiB/s to 2.6MiB/s reading from the NFS4-share on Gbit-Lan 2) Mounting with -t newnfs -o nfsv3 results in no performance gain

Re: Why is NFSv4 so slow?

2010-09-01 Thread Rick Macklem
Hi everyone, I am experiencing similar issues with newnfs: 1) I have two clients that each get around 0.5MiB/s to 2.6MiB/s reading from the NFS4-share on Gbit-Lan 2) Mounting with -t newnfs -o nfsv3 results in no performance gain whatsoever. 3) Mounting with -t nfs results in

Re: Why is NFSv4 so slow?

2010-08-30 Thread Rick C. Petty
On Sun, Aug 29, 2010 at 11:44:06AM -0400, Rick Macklem wrote: Hi. I'm still having problems with NFSv4 being very laggy on one client. When the NFSv4 server is at 50% idle CPU and the disks are 1% busy, I am getting horrible throughput on an idle client. Using dd(1) with 1 MB block

Re: Why is NFSv4 so slow?

2010-08-30 Thread Rick Macklem
Well I wouldn't say well. Every client I've set up has had this issue, and somehow through tweaking various settings and restarting nfs a bunch of times, I've been able to make it tolerable for most clients. Only one client is behaving well, and that happens to be the only machine I

Re: Why is NFSv4 so slow?

2010-08-30 Thread Rick Macklem
On Sun, Aug 29, 2010 at 11:44:06AM -0400, Rick Macklem wrote: Hi. I'm still having problems with NFSv4 being very laggy on one client. When the NFSv4 server is at 50% idle CPU and the disks are 1% busy, I am getting horrible throughput on an idle client. Using dd(1) with 1

Re: Why is NFSv4 so slow?

2010-08-29 Thread Rick Macklem
Hi. I'm still having problems with NFSv4 being very laggy on one client. When the NFSv4 server is at 50% idle CPU and the disks are 1% busy, I am getting horrible throughput on an idle client. Using dd(1) with 1 MB block size, when I try to read a 100 MB file from the client, I'm getting

Re: Why is NFSv4 so slow?

2010-08-28 Thread Rick C. Petty
Hi. I'm still having problems with NFSv4 being very laggy on one client. When the NFSv4 server is at 50% idle CPU and the disks are 1% busy, I am getting horrible throughput on an idle client. Using dd(1) with 1 MB block size, when I try to read a 100 MB file from the client, I'm getting

Re: Why is NFSv4 so slow?

2010-07-05 Thread Rick Macklem
On Sun, 27 Jun 2010, Rick C. Petty wrote: First off, many thanks to Rick Macklem for making NFSv4 possible in FreeBSD! I recently updated my NFS server and clients to v4, but have since noticed significant performance penalties. For instance, when I try ls a b c (if a, b, and c are empty

Re: Why is NFSv4 so slow?

2010-07-01 Thread Rick Macklem
On Mon, 28 Jun 2010, Rick C. Petty wrote: On Mon, Jun 28, 2010 at 12:35:14AM -0400, Rick Macklem wrote: Being stuck in newnfsreq means that it is trying to establish a TCP connection with the server (again smells like some networking issue). snip Disabling delegations is the next step.

Re: Why is NFSv4 so slow? (root/toor)

2010-06-30 Thread Rick Macklem
On Wed, 30 Jun 2010, Ian Smith wrote: I wondered whether this might be a Linux thing. On my 7.2 system, % find /usr/src -name *.[ch] -exec grep -Hw getpwuid {} \; file returns 195 lines, many in the form getpwuid(getuid()), in many base and contrib components - including id(1), bind,

Re: Why is NFSv4 so slow? (root/toor)

2010-06-29 Thread Rick Macklem
On Tue, 29 Jun 2010, Ian Smith wrote: Not wanting to hijack this (interesting) thread, but .. I have to concur with Rick P - that's rather a odd requirement when each FreeBSD install since at least 2.2 has come with root and toor (in that order) in /etc/passwd. I don't use toor, but often

Re: Why is NFSv4 so slow? (root/toor)

2010-06-29 Thread Adam Vande More
On Tue, Jun 29, 2010 at 9:58 AM, Rick Macklem rmack...@uoguelph.ca wrote: I suppose if the FreeBSD world feels that root and toor must both exist in the password database, then nfsuserd could be hacked to handle the case of translating uid 0 to root without calling getpwuid(). It seems ugly,

Re: Why is NFSv4 so slow? (root/toor)

2010-06-29 Thread Rick C. Petty
On Tue, Jun 29, 2010 at 10:20:57AM -0500, Adam Vande More wrote: On Tue, Jun 29, 2010 at 9:58 AM, Rick Macklem rmack...@uoguelph.ca wrote: I suppose if the FreeBSD world feels that root and toor must both exist in the password database, then nfsuserd could be hacked to handle the case of

Re: Why is NFSv4 so slow? (root/toor)

2010-06-29 Thread Rick Macklem
On Tue, 29 Jun 2010, Rick C. Petty wrote: To be fair, I'm not sure this is even a problem. Rick M. only suggested it as a possibility. I would think that getpwuid() would return the first match which has always been root. At least that's what it does when scanning the passwd file; I'm not

Re: Why is NFSv4 so slow?

2010-06-29 Thread Rick Macklem
On Mon, 28 Jun 2010, Rick C. Petty wrote: It would be interesting to see if the performance problem exists for NFSv3 mounts against the experimental (nfsv4) server. Hmm, I couldn't reproduce the problem. Once I unmounted the nfsv4 client and tried v3, the jittering stopped. Then I

Re: Why is NFSv4 so slow? (root/toor)

2010-06-29 Thread Dan Nelson
In the last episode (Jun 29), Rick C. Petty said: On Tue, Jun 29, 2010 at 10:20:57AM -0500, Adam Vande More wrote: On Tue, Jun 29, 2010 at 9:58 AM, Rick Macklem rmack...@uoguelph.ca wrote: I suppose if the FreeBSD world feels that root and toor must both exist in the password database,

Re: Why is NFSv4 so slow? (root/toor)

2010-06-29 Thread Ian Smith
On Tue, 29 Jun 2010, Dan Nelson wrote: In the last episode (Jun 29), Rick C. Petty said: On Tue, Jun 29, 2010 at 10:20:57AM -0500, Adam Vande More wrote: On Tue, Jun 29, 2010 at 9:58 AM, Rick Macklem rmack...@uoguelph.ca wrote: I suppose if the FreeBSD world feels that root

Re: Why is NFSv4 so slow?

2010-06-28 Thread Rick C. Petty
On Mon, Jun 28, 2010 at 12:30:30AM -0400, Rick Macklem wrote: I can't explain the corruption, beyond the fact that soft,intr can cause all sorts of grief. If mounts without soft,intr still show corruption problems, try disabling delegations (either kill off the nfscbd daemons on the client

Re: Why is NFSv4 so slow?

2010-06-28 Thread Rick C. Petty
On Sun, Jun 27, 2010 at 09:58:53PM -0700, Jeremy Chadwick wrote: Again, my ports tree is mounted as FSType nfs with option nfsv4. FreeBSD/amd64 8.1-PRERELEASE r208408M GENERIC kernel. This sounds like NFSv4 is tickling some kind of bug in your NIC driver but I'm not entirely sure. Can

Re: Why is NFSv4 so slow?

2010-06-28 Thread Jeremy Chadwick
On Mon, Jun 28, 2010 at 09:20:25AM -0500, Rick C. Petty wrote: Again, my ports tree is mounted as FSType nfs with option nfsv4. FreeBSD/amd64 8.1-PRERELEASE r208408M GENERIC kernel. This sounds like NFSv4 is tickling some kind of bug in your NIC driver but I'm not entirely sure.

Re: Why is NFSv4 so slow?

2010-06-28 Thread Rick C. Petty
On Mon, Jun 28, 2010 at 07:56:00AM -0700, Jeremy Chadwick wrote: Three other things to provide output from if you could (you can X out IPs and MACs too), from both client and server: 6) netstat -idn server: NameMtu Network Address Ipkts Ierrs IdropOpkts Oerrs

Re: Why is NFSv4 so slow?

2010-06-28 Thread Rick C. Petty
On Mon, Jun 28, 2010 at 12:35:14AM -0400, Rick Macklem wrote: Being stuck in newnfsreq means that it is trying to establish a TCP connection with the server (again smells like some networking issue). snip Disabling delegations is the next step. (They aren't required for correct behaviour

Re: Why is NFSv4 so slow?

2010-06-28 Thread Jeremy Chadwick
On Mon, Jun 28, 2010 at 10:18:35AM -0500, Rick C. Petty wrote: 8) Contents of /etc/sysctl.conf server and client: # for NFSv4 kern.ipc.maxsockbuf=524288 You might want to discuss this one with Rick a bit (I'm not sure of the implications). Regarding heavy network I/O (I don't use NFS

Re: Why is NFSv4 so slow?

2010-06-28 Thread Rick Macklem
On Mon, 28 Jun 2010, Rick C. Petty wrote: On Mon, Jun 28, 2010 at 12:35:14AM -0400, Rick Macklem wrote: Being stuck in newnfsreq means that it is trying to establish a TCP connection with the server (again smells like some networking issue). snip Disabling delegations is the next step.

Re: Why is NFSv4 so slow?

2010-06-28 Thread Rick Macklem
On Mon, 28 Jun 2010, Rick C. Petty wrote: Make sure you don't have multiple entries for the same uid, such as root and toor both for uid 0 in your /etc/passwd. (ie. get rid of one of them, if you have both) Hmm, that's a strange requirement, since FreeBSD by default comes with both. That

Re: Why is NFSv4 so slow?

2010-06-28 Thread Rick Macklem
On Mon, 28 Jun 2010, Rick C. Petty wrote: I can try it again with v3 client and v4 server, if you think that's worthy of pursuit. If it makes any difference, the server's four CPUs are pegged at 100% (running nice +4 cpu-bound jobs). But that was the case before I enabled v4 server too.

Re: Why is NFSv4 so slow?

2010-06-28 Thread Rick C. Petty
On Mon, Jun 28, 2010 at 10:09:21PM -0400, Rick Macklem wrote: On Mon, 28 Jun 2010, Rick C. Petty wrote: If it makes any difference, the server's four CPUs are pegged at 100% (running nice +4 cpu-bound jobs). But that was the case before I enabled v4 server too. If it is practical, it

Re: Why is NFSv4 so slow?

2010-06-28 Thread Rick C. Petty
On Mon, Jun 28, 2010 at 09:29:11AM -0700, Jeremy Chadwick wrote: # Increase send/receive buffer maximums from 256KB to 16MB. # FreeBSD 7.x and later will auto-tune the size, but only up to the max. net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 # Double send/receive

Re: Why is NFSv4 so slow?

2010-06-28 Thread Rick C. Petty
On Mon, Jun 28, 2010 at 07:48:59PM -0400, Rick Macklem wrote: Ok, it sounds like you found some kind of race condition in the delegation handling. (I'll see if I can reproduce it here. It could be fun to find:-) Good luck with that! =) I can try it again with v3 client and v4 server, if

Re: Why is NFSv4 so slow? (root/toor)

2010-06-28 Thread Ian Smith
On Mon, 28 Jun 2010, Rick Macklem wrote: On Mon, 28 Jun 2010, Rick C. Petty wrote: Make sure you don't have multiple entries for the same uid, such as root and toor both for uid 0 in your /etc/passwd. (ie. get rid of one of them, if you have both) Hmm, that's a

Re: Why is NFSv4 so slow?

2010-06-27 Thread Rick Macklem
On Sun, 27 Jun 2010, Rick C. Petty wrote: First off, many thanks to Rick Macklem for making NFSv4 possible in FreeBSD! I recently updated my NFS server and clients to v4, but have since noticed significant performance penalties. For instance, when I try ls a b c (if a, b, and c are empty

Re: Why is NFSv4 so slow?

2010-06-27 Thread Rick C. Petty
On Sun, Jun 27, 2010 at 08:04:28PM -0400, Rick Macklem wrote: Weird, I don't see that here. The only thing I can think of is that the experimental client/server will try to do I/O at the size of MAXBSIZE by default, which might be causing a burst of traffic your net interface can't keep up

Re: Why is NFSv4 so slow?

2010-06-27 Thread Rick C. Petty
On Sun, Jun 27, 2010 at 08:04:28PM -0400, Rick Macklem wrote: Weird, I don't see that here. The only thing I can think of is that the experimental client/server will try to do I/O at the size of MAXBSIZE by default, which might be causing a burst of traffic your net interface can't keep up

Re: Why is NFSv4 so slow?

2010-06-27 Thread Rick Macklem
On Sun, 27 Jun 2010, Rick C. Petty wrote: Hmm. When I mounted the same filesystem with nfs3 from a different client, everything started working at almost normal speed (still a little slower though). Now on that same host I saw a file get corrupted. On the server, I see the following: %

Re: Why is NFSv4 so slow?

2010-06-27 Thread Rick Macklem
On Sun, 27 Jun 2010, Rick C. Petty wrote: On Sun, Jun 27, 2010 at 08:04:28PM -0400, Rick Macklem wrote: Weird, I don't see that here. The only thing I can think of is that the experimental client/server will try to do I/O at the size of MAXBSIZE by default, which might be causing a burst of

Re: Why is NFSv4 so slow?

2010-06-27 Thread Jeremy Chadwick
On Sun, Jun 27, 2010 at 10:47:41PM -0500, Rick C. Petty wrote: On Sun, Jun 27, 2010 at 08:04:28PM -0400, Rick Macklem wrote: Weird, I don't see that here. The only thing I can think of is that the experimental client/server will try to do I/O at the size of MAXBSIZE by default, which