nfs problems
Hi, starting about last week, I'm getting: rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32) rsync: write failed on /net/rnd/dist/tmp/local/amd64.FreeBSD_8.3-wip/compat/li nux/usr/lib/locale/locale-archive.tmpl: Permission denied (13) rsync error: error in file IO (code 11) at receiver.c(322) [receiver=3.0.9] rsync: connection unexpectedly closed (21872 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9] the server is running 8.2, but the client is very upto date, 8.3-stable as of this morning (local time). after runing rsync several times, it finaly gets synced. another item is that i'm using am-utils, but I don't see it causing the problem I will try using tcp (instead of udp) soon. any insights? cheers, danny ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: nfs problems
Hi, starting about last week, I'm getting: rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32) rsync: write failed on /net/rnd/dist/tmp/local/amd64.FreeBSD_8.3-wip/compat/li nux/usr/lib/locale/locale-archive.tmpl: Permission denied (13) rsync error: error in file IO (code 11) at receiver.c(322) [receiver=3.0.9] rsync: connection unexpectedly closed (21872 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9] the server is running 8.2, but the client is very upto date, 8.3-stable as of this morning (local time). after runing rsync several times, it finaly gets synced. another item is that i'm using am-utils, but I don't see it causing the problem I will try using tcp (instead of udp) soon. any insights? cheers, danny the problem is most probably NFS/UDP related. I took am-utils out of the equation. mounted using TCP, and no problems mounted using UDP: Jun 29 12:38:14 pe-02 kernel: nfs server nrnfdn:sf/s ds isseterr:vve ernr o trrnn dd:r:e/s/pddoinisdsitt::n nngoo Jun 29 12:38:14 pe-02 kernel: tt Jun 29 12:38:14 pe-02 kernel: Jun 29 12:38:14 pe-02 kernel: 66 rreessppoonnddiinngg Jun 29 12:38:14 pe-02 kernel: Jun 29 12:38:14 pe-02 kernel: nfs server rnd:/dist: not responding Jun 29 12:38:14 pe-02 last message repeated 11 times Jun 29 12:38:27 pe-02 kernel: nfs server rnd:/dist: is alive again the above happens about every 15 seconds (you have to learn to read in between the bytes :-) cheers, danny ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: nfs problems
On 29/06/2012 10:45, Daniel Braniss wrote: Hi, starting about last week, I'm getting: rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32) rsync: write failed on /net/rnd/dist/tmp/local/amd64.FreeBSD_8.3-wip/compat/li nux/usr/lib/locale/locale-archive.tmpl: Permission denied (13) rsync error: error in file IO (code 11) at receiver.c(322) [receiver=3.0.9] rsync: connection unexpectedly closed (21872 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9] the server is running 8.2, but the client is very upto date, 8.3-stable as of this morning (local time). after runing rsync several times, it finaly gets synced. another item is that i'm using am-utils, but I don't see it causing the problem I will try using tcp (instead of udp) soon. any insights? cheers, danny the problem is most probably NFS/UDP related. I took am-utils out of the equation. mounted using TCP, and no problems mounted using UDP: Jun 29 12:38:14 pe-02 kernel: nfs server nrnfdn:sf/s ds isseterr:vve ernr o trrnn dd:r:e/s/pddoinisdsitt::n nngoo Jun 29 12:38:14 pe-02 kernel: tt Jun 29 12:38:14 pe-02 kernel: Jun 29 12:38:14 pe-02 kernel: 66 rreessppoonnddiinngg Jun 29 12:38:14 pe-02 kernel: Jun 29 12:38:14 pe-02 kernel: nfs server rnd:/dist: not responding Jun 29 12:38:14 pe-02 last message repeated 11 times Jun 29 12:38:27 pe-02 kernel: nfs server rnd:/dist: is alive again the above happens about every 15 seconds (you have to learn to read in between the bytes :-) cheers, danny Its also possible you are hitting a bug I came across recently. See http://lists.freebsd.org/pipermail/freebsd-current/2012-June/034860.html basicly mountd may give incorrect permission denied errors when it is refreshing the exports list due to non-atomic operations. see kern/131342 kern/136865 also. Vince ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: nfs problems
On 29/06/2012 10:45, Daniel Braniss wrote: Hi, starting about last week, I'm getting: rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32) rsync: write failed on /net/rnd/dist/tmp/local/amd64.FreeBSD_8.3-wip/compat/li nux/usr/lib/locale/locale-archive.tmpl: Permission denied (13) rsync error: error in file IO (code 11) at receiver.c(322) [receiver=3.0.9] rsync: connection unexpectedly closed (21872 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9] the server is running 8.2, but the client is very upto date, 8.3-stable as of this morning (local time). after runing rsync several times, it finaly gets synced. another item is that i'm using am-utils, but I don't see it causing the problem I will try using tcp (instead of udp) soon. any insights? cheers, danny the problem is most probably NFS/UDP related. I took am-utils out of the equation. mounted using TCP, and no problems mounted using UDP: Jun 29 12:38:14 pe-02 kernel: nfs server nrnfdn:sf/s ds isseterr:vve ernr o trrnn dd:r:e/s/pddoinisdsitt::n nngoo Jun 29 12:38:14 pe-02 kernel: tt Jun 29 12:38:14 pe-02 kernel: Jun 29 12:38:14 pe-02 kernel: 66 rreessppoonnddiinngg Jun 29 12:38:14 pe-02 kernel: Jun 29 12:38:14 pe-02 kernel: nfs server rnd:/dist: not responding Jun 29 12:38:14 pe-02 last message repeated 11 times Jun 29 12:38:27 pe-02 kernel: nfs server rnd:/dist: is alive again the above happens about every 15 seconds (you have to learn to read in between the bytes :-) cheers, danny Its also possible you are hitting a bug I came across recently. See http://lists.freebsd.org/pipermail/freebsd-current/2012-June/034860.html basicly mountd may give incorrect permission denied errors when it is refreshing the exports list due to non-atomic operations. see kern/131342 kern/136865 Hi Vince, I thought so too, there used to be a bug caused by am-utils umounting, succeding even if the mount was active, then re-mounting, which caused all kind of problems, the work around was to increase the timeout. But I don't think it's the case here, unless mountd has a life of its own. Furthermore, rsync works without a glitch when mounted nfs/tcp. thanks, danny ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: FreeBSD 7.0-STABLE: mount_nfs vs mount -t nfs: problems with second one, UDP timeouts and ICMP ports unreach?!
Lev Serebryakov wrote: You see? b answer with UDP port unreachable on each RPC reply! Additional info. ktrace from mount -t nfs: = 65962 mount_nfs 0.006679 RET sendto 40/0x28 65962 mount_nfs 0.006682 CALL kevent(0x4,0x638110,0x1,0x7fffe2d0,0x1,0x7fffe310) 65962 mount_nfs 10.218001 GIO fd 4 wrote 32 bytes 65962 mount_nfs 10.218018 GIO fd 4 read 0 bytes 65962 mount_nfs 10.218023 RET kevent 0 65962 mount_nfs 10.218029 CALL gettimeofday(0x7fffe320,0) 65962 mount_nfs 10.218035 RET gettimeofday 0 65962 mount_nfs 10.218040 CALL close(0x4) 65962 mount_nfs 10.218049 RET close 0 65962 mount_nfs 10.218077 CALL close(0x3) 65962 mount_nfs 10.218087 RET close 0 65962 mount_nfs 10.218101 CALL write(0x2,0x7fffdd40,0x37) 65962 mount_nfs 10.218117 GIO fd 2 wrote 55 bytes [udp] gateway:/usr/ports: NFSPROC_NULL: RPC: Timed out = ktrace (same piece) from mount_nfs: = 93109 mount_nfs 0.005458 RET sendto 40/0x28 93109 mount_nfs 0.005462 CALL kevent(0x4,0x638110,0x1,0x7fffe2a0,0x1,0x7fffe2e0) 93109 mount_nfs 0.005724 GIO fd 4 wrote 32 bytes 93109 mount_nfs 0.005738 GIO fd 4 read 32 bytes 93109 mount_nfs 0.005743 RET kevent 1 93109 mount_nfs 0.005748 CALL recvfrom(0x3,0x638134,0x2260,0,0,0) 93109 mount_nfs 0.005756 GIO fd 3 read 24 bytes 93109 mount_nfs 0.005761 RET recvfrom 24/0x18 93109 mount_nfs 0.005767 CALL close(0x4) 93109 mount_nfs 0.005775 RET close 0 93109 mount_nfs 0.005781 CALL close(0x3) 93109 mount_nfs 0.005788 RET close 0 = ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 7.0-STABLE: mount_nfs vs mount -t nfs: problems with second one, UDP timeouts and ICMP ports unreach?!
Lev Serebryakov wrote: Main problem is, that /etc/fstab is processed by mount, and NFS mount hangs up on boot, as shown above :( Mounting with mount -t nfs with 7.0 server (host B) and 6.3 client (host A) works... -- // Lev Serebryakov ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 7.0-STABLE: mount_nfs vs mount -t nfs: problems with second one, UDP timeouts and ICMP ports unreach?!
Hello, freebsd-stable. You wrote 16 мая 2008 г., 14:40:18: There is NO any firewalls on B. And, I repeat, it WORKS when I call mount_nfs directly in a moment! Adding `-o -c' to mount (to pass `-c' to mount_nfs) helps. But I'm very curious WHY mount_nfs, called directly, work WITHOUT `-c'... -- // Black Lion AKA Lev Serebryakov [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
NFS problems
I upgraded 9 of my systems to RELENG_5 on Oct 29 and 30. Now none of them can do a dump to an NFS mounted directory. the NFS connection is made, because the dump file is created on the NFS directory, but it stays at 0 bytes. The system that is doing the dump hangs after: oregon root:#dump -0auf - usr /host/backup.etv.net/back/oregon.etv.net._usr.2005-11-03.level-0 DUMP: WARNING: should use -L when dumping live read-write filesystems! DUMP: Date of this level 0 dump: Thu Nov 3 09:25:44 2005 DUMP: Date of last level 0 dump: the epoch DUMP: Dumping /dev/ad12s1d (/usr) to standard output DUMP: mapping (Pass I) [regular files] DUMP: mapping (Pass II) [directories] DUMP: estimated 10647090 tape blocks. Any suggestions/pointers/etc would be welcome. TIA Elliot ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: NFS problems
- Original Message - From: Elliot Finley [EMAIL PROTECTED] I upgraded 9 of my systems to RELENG_5 on Oct 29 and 30. Now none of them can do a dump to an NFS mounted directory. Oops I also changed some ipf rules and after opening everything up, the dump works again. Sorry for the noise. On another note: Now I'm not sure how to make dumps work and have ipf functioning. in ipf.rules of the server receiving the dump I have: block in all block out all pass out quick on lo0 pass in quick on lo0 pass out quick from myIP/32 to any keep state pass in quick from mySubnet/27 to any keep state and both the sending and receiving server are on mySubnet. What am I missing? TIA Elliot ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Weird NFS problems
Jon Dama wrote: Try switching to TCP NFS. a 100MBit interface cannot keep up with a 1GBit interface in a bridge configuration. Therefore, in the long run, at full-bore you'd expect to drop 9 out of every 10 ethernet frames. MTU is 1500 therefore 1K works (it fits in one frame), 2K doesn't (your NFS transactions are split across frames, one of which will almost certainly be dropped, it's UDP so the loss of one frame invalidates the whole transaction). This is the same reason you can't use UDP with a block size greater than MTU to use NFS over your DSL or some such arrangement. Incidentially, this has nothing to do with FreeBSD. So if using TCP mounts solves your problem, don't expect Solaris NFS to magically make the UDP case work... The thing is that UDP NFS has been working for us for years. A big part of our work is performance analysis, so to change our network architecture will invalidate a large part of our data. -- -- Skylar Thompson ([EMAIL PROTECTED]) -- http://www.cs.earlham.edu/~skylar/ signature.asc Description: OpenPGP digital signature
Re: Weird NFS problems
Jon Dama wrote: Try switching to TCP NFS. a 100MBit interface cannot keep up with a 1GBit interface in a bridge configuration. Therefore, in the long run, at full-bore you'd expect to drop 9 out of every 10 ethernet frames. MTU is 1500 therefore 1K works (it fits in one frame), 2K doesn't (your NFS transactions are split across frames, one of which will almost certainly be dropped, it's UDP so the loss of one frame invalidates the whole transaction). This is the same reason you can't use UDP with a block size greater than MTU to use NFS over your DSL or some such arrangement. Incidentially, this has nothing to do with FreeBSD. So if using TCP mounts solves your problem, don't expect Solaris NFS to magically make the UDP case work... The thing is that UDP NFS has been working for us for years. A big part of our work is performance analysis, so to change our network architecture will invalidate a large part of our data. -- -- Skylar Thompson ([EMAIL PROTECTED]) -- http://www.cs.earlham.edu/~skylar/ signature.asc Description: OpenPGP digital signature
Re: Weird NFS problems
Yes, but surely you weren't bridging gigabit and 100Mbit before? Did you try my suggestion about binding the IP address of the NFS server to the 100Mbit side? -Jon On Tue, 31 May 2005, Skylar Thompson wrote: Jon Dama wrote: Try switching to TCP NFS. a 100MBit interface cannot keep up with a 1GBit interface in a bridge configuration. Therefore, in the long run, at full-bore you'd expect to drop 9 out of every 10 ethernet frames. MTU is 1500 therefore 1K works (it fits in one frame), 2K doesn't (your NFS transactions are split across frames, one of which will almost certainly be dropped, it's UDP so the loss of one frame invalidates the whole transaction). This is the same reason you can't use UDP with a block size greater than MTU to use NFS over your DSL or some such arrangement. Incidentially, this has nothing to do with FreeBSD. So if using TCP mounts solves your problem, don't expect Solaris NFS to magically make the UDP case work... The thing is that UDP NFS has been working for us for years. A big part of our work is performance analysis, so to change our network architecture will invalidate a large part of our data. -- -- Skylar Thompson ([EMAIL PROTECTED]) -- http://www.cs.earlham.edu/~skylar/ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Weird NFS problems
Yes, but surely you weren't bridging gigabit and 100Mbit before? Did you try my suggestion about binding the IP address of the NFS server to the 100Mbit side? -Jon On Tue, 31 May 2005, Skylar Thompson wrote: Jon Dama wrote: Try switching to TCP NFS. a 100MBit interface cannot keep up with a 1GBit interface in a bridge configuration. Therefore, in the long run, at full-bore you'd expect to drop 9 out of every 10 ethernet frames. MTU is 1500 therefore 1K works (it fits in one frame), 2K doesn't (your NFS transactions are split across frames, one of which will almost certainly be dropped, it's UDP so the loss of one frame invalidates the whole transaction). This is the same reason you can't use UDP with a block size greater than MTU to use NFS over your DSL or some such arrangement. Incidentially, this has nothing to do with FreeBSD. So if using TCP mounts solves your problem, don't expect Solaris NFS to magically make the UDP case work... The thing is that UDP NFS has been working for us for years. A big part of our work is performance analysis, so to change our network architecture will invalidate a large part of our data. -- -- Skylar Thompson ([EMAIL PROTECTED]) -- http://www.cs.earlham.edu/~skylar/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Weird NFS problems
Jon Dama wrote: Yes, but surely you weren't bridging gigabit and 100Mbit before? Did you try my suggestion about binding the IP address of the NFS server to the 100Mbit side? Yeah. Unfortunately networking on the server fell apart when I did that. Traffic was still passed and I could get through to the server on the 100Mb/s side, but not on the 1000Mb/s. It looked like the arp tables weren't being forwarded properly, but I couldn't convince FreeBSD to do proxy arp. After doing some more poking around, it actually looks like it might be a misfeature in the Linux 2.4 kernel wrt ipfilter (which is running on the bridge). Apparently 2.4 fragments UDP packets in the reverse order that every other UNIX-like operating system does, which throws off ipfilter's state tables. I'm going to do some testing to see if the difference between UDP and TCP NFS is negligible enough for us to disregard. Thanks for the suggestions! -- -- Skylar Thompson ([EMAIL PROTECTED]) -- http://www.cs.earlham.edu/~skylar/ signature.asc Description: OpenPGP digital signature
Re: Weird NFS problems
Jon Dama wrote: Yes, but surely you weren't bridging gigabit and 100Mbit before? Did you try my suggestion about binding the IP address of the NFS server to the 100Mbit side? Yeah. Unfortunately networking on the server fell apart when I did that. Traffic was still passed and I could get through to the server on the 100Mb/s side, but not on the 1000Mb/s. It looked like the arp tables weren't being forwarded properly, but I couldn't convince FreeBSD to do proxy arp. After doing some more poking around, it actually looks like it might be a misfeature in the Linux 2.4 kernel wrt ipfilter (which is running on the bridge). Apparently 2.4 fragments UDP packets in the reverse order that every other UNIX-like operating system does, which throws off ipfilter's state tables. I'm going to do some testing to see if the difference between UDP and TCP NFS is negligible enough for us to disregard. Thanks for the suggestions! -- -- Skylar Thompson ([EMAIL PROTECTED]) -- http://www.cs.earlham.edu/~skylar/ signature.asc Description: OpenPGP digital signature
Re: Weird NFS problems
Oh, something else to try: I checked through my notes and discovered that I had gotten UDP to work in a similar configuration before. What I did was bind the IP address to fxp0 instead of em0. By doing this, the kernel seems to send the data at a pace suitable for the slow interface. -Jon On Fri, 27 May 2005, Don Lewis wrote: On 26 May, Skylar Thompson wrote: I'm having some problems with NFS serving on a FreeBSD 5.4-RELEASE machine. The FreeBSD machine is the NFS/NIS server for a group of four Linux clusters. The network archictecture looks like this: 234/24 234/24 Cluster 1 ---|--- Cluster 3 | --- em0| File server | fxp0 | -- Cluster 2 ---|--- Cluster 4 234/24230/24 em0 and fxp0 are bridged, and em0 has a 234/24 IP address while fxp0 is just in promiscuous mode. 234/24 is an 802.1q VLAN on the fxp0 side of the server, so packets are untagged at the switch just before fxp0, and are forwarded to em0 through the bridge. The problem manifests itself in large UDP NFS requests from Clusters 3 and 4. The export can be mounted fine from both those clusters, and small transfers such as with ls work fine, but the moment any serious data transfer starts, the entire mount just hangs. Running ethereal on the file server shows a a lot of fragmented packets, and RPC retransmissions on just a single request. Reducing the read and write NFS buffers on the Linux clients to 1kB from the default of 4kB solves the issue, but kills the transfer rate. The moment I go to 2kB, the problem reappearss. Clusters 1 and 2 use the default of 4kB buffers, and have no problems communicating to em0. Poking through the list archives, I ran across this message (http://lists.freebsd.org/pipermail/freebsd-stable/2003-May/001007.html) that reveals a bug in the fxp(4) driver in 4-RELEASE that incorrectly detects the capabilities of the NIC. Is this still an issue in 5-RELEASE, or am I looking at a different problem? Any ideas on how I can get the NFS buffers up to a reasonable level? That problem was fixed quite some time ago. Which transfer direction fails? Client writing to server Client reading from server Both? Do you see all the fragments in the retransmitted request? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Weird NFS problems
Oh, something else to try: I checked through my notes and discovered that I had gotten UDP to work in a similar configuration before. What I did was bind the IP address to fxp0 instead of em0. By doing this, the kernel seems to send the data at a pace suitable for the slow interface. -Jon On Fri, 27 May 2005, Don Lewis wrote: On 26 May, Skylar Thompson wrote: I'm having some problems with NFS serving on a FreeBSD 5.4-RELEASE machine. The FreeBSD machine is the NFS/NIS server for a group of four Linux clusters. The network archictecture looks like this: 234/24 234/24 Cluster 1 ---|--- Cluster 3 | --- em0| File server | fxp0 | -- Cluster 2 ---|--- Cluster 4 234/24230/24 em0 and fxp0 are bridged, and em0 has a 234/24 IP address while fxp0 is just in promiscuous mode. 234/24 is an 802.1q VLAN on the fxp0 side of the server, so packets are untagged at the switch just before fxp0, and are forwarded to em0 through the bridge. The problem manifests itself in large UDP NFS requests from Clusters 3 and 4. The export can be mounted fine from both those clusters, and small transfers such as with ls work fine, but the moment any serious data transfer starts, the entire mount just hangs. Running ethereal on the file server shows a a lot of fragmented packets, and RPC retransmissions on just a single request. Reducing the read and write NFS buffers on the Linux clients to 1kB from the default of 4kB solves the issue, but kills the transfer rate. The moment I go to 2kB, the problem reappearss. Clusters 1 and 2 use the default of 4kB buffers, and have no problems communicating to em0. Poking through the list archives, I ran across this message (http://lists.freebsd.org/pipermail/freebsd-stable/2003-May/001007.html) that reveals a bug in the fxp(4) driver in 4-RELEASE that incorrectly detects the capabilities of the NIC. Is this still an issue in 5-RELEASE, or am I looking at a different problem? Any ideas on how I can get the NFS buffers up to a reasonable level? That problem was fixed quite some time ago. Which transfer direction fails? Client writing to server Client reading from server Both? Do you see all the fragments in the retransmitted request? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Weird NFS problems
On 26 May, Skylar Thompson wrote: I'm having some problems with NFS serving on a FreeBSD 5.4-RELEASE machine. The FreeBSD machine is the NFS/NIS server for a group of four Linux clusters. The network archictecture looks like this: 234/24 234/24 Cluster 1 ---|--- Cluster 3 | --- em0| File server | fxp0 | -- Cluster 2 ---|--- Cluster 4 234/24230/24 em0 and fxp0 are bridged, and em0 has a 234/24 IP address while fxp0 is just in promiscuous mode. 234/24 is an 802.1q VLAN on the fxp0 side of the server, so packets are untagged at the switch just before fxp0, and are forwarded to em0 through the bridge. The problem manifests itself in large UDP NFS requests from Clusters 3 and 4. The export can be mounted fine from both those clusters, and small transfers such as with ls work fine, but the moment any serious data transfer starts, the entire mount just hangs. Running ethereal on the file server shows a a lot of fragmented packets, and RPC retransmissions on just a single request. Reducing the read and write NFS buffers on the Linux clients to 1kB from the default of 4kB solves the issue, but kills the transfer rate. The moment I go to 2kB, the problem reappearss. Clusters 1 and 2 use the default of 4kB buffers, and have no problems communicating to em0. Poking through the list archives, I ran across this message (http://lists.freebsd.org/pipermail/freebsd-stable/2003-May/001007.html) that reveals a bug in the fxp(4) driver in 4-RELEASE that incorrectly detects the capabilities of the NIC. Is this still an issue in 5-RELEASE, or am I looking at a different problem? Any ideas on how I can get the NFS buffers up to a reasonable level? That problem was fixed quite some time ago. Which transfer direction fails? Client writing to server Client reading from server Both? Do you see all the fragments in the retransmitted request? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Weird NFS problems
On 26 May, Skylar Thompson wrote: I'm having some problems with NFS serving on a FreeBSD 5.4-RELEASE machine. The FreeBSD machine is the NFS/NIS server for a group of four Linux clusters. The network archictecture looks like this: 234/24 234/24 Cluster 1 ---|--- Cluster 3 | --- em0| File server | fxp0 | -- Cluster 2 ---|--- Cluster 4 234/24230/24 em0 and fxp0 are bridged, and em0 has a 234/24 IP address while fxp0 is just in promiscuous mode. 234/24 is an 802.1q VLAN on the fxp0 side of the server, so packets are untagged at the switch just before fxp0, and are forwarded to em0 through the bridge. The problem manifests itself in large UDP NFS requests from Clusters 3 and 4. The export can be mounted fine from both those clusters, and small transfers such as with ls work fine, but the moment any serious data transfer starts, the entire mount just hangs. Running ethereal on the file server shows a a lot of fragmented packets, and RPC retransmissions on just a single request. Reducing the read and write NFS buffers on the Linux clients to 1kB from the default of 4kB solves the issue, but kills the transfer rate. The moment I go to 2kB, the problem reappearss. Clusters 1 and 2 use the default of 4kB buffers, and have no problems communicating to em0. Poking through the list archives, I ran across this message (http://lists.freebsd.org/pipermail/freebsd-stable/2003-May/001007.html) that reveals a bug in the fxp(4) driver in 4-RELEASE that incorrectly detects the capabilities of the NIC. Is this still an issue in 5-RELEASE, or am I looking at a different problem? Any ideas on how I can get the NFS buffers up to a reasonable level? That problem was fixed quite some time ago. Which transfer direction fails? Client writing to server Client reading from server Both? Do you see all the fragments in the retransmitted request? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Weird NFS problems
Try switching to TCP NFS. a 100MBit interface cannot keep up with a 1GBit interface in a bridge configuration. Therefore, in the long run, at full-bore you'd expect to drop 9 out of every 10 ethernet frames. MTU is 1500 therefore 1K works (it fits in one frame), 2K doesn't (your NFS transactions are split across frames, one of which will almost certainly be dropped, it's UDP so the loss of one frame invalidates the whole transaction). This is the same reason you can't use UDP with a block size greater than MTU to use NFS over your DSL or some such arrangement. Incidentially, this has nothing to do with FreeBSD. So if using TCP mounts solves your problem, don't expect Solaris NFS to magically make the UDP case work... -Jon On Fri, 27 May 2005, Don Lewis wrote: On 26 May, Skylar Thompson wrote: I'm having some problems with NFS serving on a FreeBSD 5.4-RELEASE machine. The FreeBSD machine is the NFS/NIS server for a group of four Linux clusters. The network archictecture looks like this: 234/24 234/24 Cluster 1 ---|--- Cluster 3 | --- em0| File server | fxp0 | -- Cluster 2 ---|--- Cluster 4 234/24230/24 em0 and fxp0 are bridged, and em0 has a 234/24 IP address while fxp0 is just in promiscuous mode. 234/24 is an 802.1q VLAN on the fxp0 side of the server, so packets are untagged at the switch just before fxp0, and are forwarded to em0 through the bridge. The problem manifests itself in large UDP NFS requests from Clusters 3 and 4. The export can be mounted fine from both those clusters, and small transfers such as with ls work fine, but the moment any serious data transfer starts, the entire mount just hangs. Running ethereal on the file server shows a a lot of fragmented packets, and RPC retransmissions on just a single request. Reducing the read and write NFS buffers on the Linux clients to 1kB from the default of 4kB solves the issue, but kills the transfer rate. The moment I go to 2kB, the problem reappearss. Clusters 1 and 2 use the default of 4kB buffers, and have no problems communicating to em0. Poking through the list archives, I ran across this message (http://lists.freebsd.org/pipermail/freebsd-stable/2003-May/001007.html) that reveals a bug in the fxp(4) driver in 4-RELEASE that incorrectly detects the capabilities of the NIC. Is this still an issue in 5-RELEASE, or am I looking at a different problem? Any ideas on how I can get the NFS buffers up to a reasonable level? That problem was fixed quite some time ago. Which transfer direction fails? Client writing to server Client reading from server Both? Do you see all the fragments in the retransmitted request? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Weird NFS problems
Try switching to TCP NFS. a 100MBit interface cannot keep up with a 1GBit interface in a bridge configuration. Therefore, in the long run, at full-bore you'd expect to drop 9 out of every 10 ethernet frames. MTU is 1500 therefore 1K works (it fits in one frame), 2K doesn't (your NFS transactions are split across frames, one of which will almost certainly be dropped, it's UDP so the loss of one frame invalidates the whole transaction). This is the same reason you can't use UDP with a block size greater than MTU to use NFS over your DSL or some such arrangement. Incidentially, this has nothing to do with FreeBSD. So if using TCP mounts solves your problem, don't expect Solaris NFS to magically make the UDP case work... -Jon On Fri, 27 May 2005, Don Lewis wrote: On 26 May, Skylar Thompson wrote: I'm having some problems with NFS serving on a FreeBSD 5.4-RELEASE machine. The FreeBSD machine is the NFS/NIS server for a group of four Linux clusters. The network archictecture looks like this: 234/24 234/24 Cluster 1 ---|--- Cluster 3 | --- em0| File server | fxp0 | -- Cluster 2 ---|--- Cluster 4 234/24230/24 em0 and fxp0 are bridged, and em0 has a 234/24 IP address while fxp0 is just in promiscuous mode. 234/24 is an 802.1q VLAN on the fxp0 side of the server, so packets are untagged at the switch just before fxp0, and are forwarded to em0 through the bridge. The problem manifests itself in large UDP NFS requests from Clusters 3 and 4. The export can be mounted fine from both those clusters, and small transfers such as with ls work fine, but the moment any serious data transfer starts, the entire mount just hangs. Running ethereal on the file server shows a a lot of fragmented packets, and RPC retransmissions on just a single request. Reducing the read and write NFS buffers on the Linux clients to 1kB from the default of 4kB solves the issue, but kills the transfer rate. The moment I go to 2kB, the problem reappearss. Clusters 1 and 2 use the default of 4kB buffers, and have no problems communicating to em0. Poking through the list archives, I ran across this message (http://lists.freebsd.org/pipermail/freebsd-stable/2003-May/001007.html) that reveals a bug in the fxp(4) driver in 4-RELEASE that incorrectly detects the capabilities of the NIC. Is this still an issue in 5-RELEASE, or am I looking at a different problem? Any ideas on how I can get the NFS buffers up to a reasonable level? That problem was fixed quite some time ago. Which transfer direction fails? Client writing to server Client reading from server Both? Do you see all the fragments in the retransmitted request? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Weird NFS problems
I'm having some problems with NFS serving on a FreeBSD 5.4-RELEASE machine. The FreeBSD machine is the NFS/NIS server for a group of four Linux clusters. The network archictecture looks like this: 234/24 234/24 Cluster 1 ---|--- Cluster 3 | --- em0| File server | fxp0 | -- Cluster 2 ---|--- Cluster 4 234/24230/24 em0 and fxp0 are bridged, and em0 has a 234/24 IP address while fxp0 is just in promiscuous mode. 234/24 is an 802.1q VLAN on the fxp0 side of the server, so packets are untagged at the switch just before fxp0, and are forwarded to em0 through the bridge. The problem manifests itself in large UDP NFS requests from Clusters 3 and 4. The export can be mounted fine from both those clusters, and small transfers such as with ls work fine, but the moment any serious data transfer starts, the entire mount just hangs. Running ethereal on the file server shows a a lot of fragmented packets, and RPC retransmissions on just a single request. Reducing the read and write NFS buffers on the Linux clients to 1kB from the default of 4kB solves the issue, but kills the transfer rate. The moment I go to 2kB, the problem reappearss. Clusters 1 and 2 use the default of 4kB buffers, and have no problems communicating to em0. Poking through the list archives, I ran across this message (http://lists.freebsd.org/pipermail/freebsd-stable/2003-May/001007.html) that reveals a bug in the fxp(4) driver in 4-RELEASE that incorrectly detects the capabilities of the NIC. Is this still an issue in 5-RELEASE, or am I looking at a different problem? Any ideas on how I can get the NFS buffers up to a reasonable level? -- -- Skylar Thompson ([EMAIL PROTECTED]) -- http://www.cs.earlham.edu/~skylar/ signature.asc Description: OpenPGP digital signature
Re: NFS Problems with Quantum Snapserver 4100 (BGE Cards!)
Already running the card and switch port in 100BaseTX FDX (forced) :) Would use GigE if the switch supported it tho Thus spake Matthew Dillon ([EMAIL PROTECTED]) : It should also work if you force the GigE card into 100BaseTX mode, assuming the switch can deal with it. Though of course then you are not getting GigE speeds. Your description is very similar to a problem I had on my 2550's that Bill Paul finally solved. It turned out that my 2550's (BCM5700) were not initializing the polynomial functions properly and this resulted in packet loss. Whenever you get bit stuffing packet loss like that, certain bit patterns will *always* fail. But I'm sure this issue has already been dealt with on the 5701 so the problem you are having is probably different. The result is the appearance of hicups and delays on a TCP connection, and repeatable hangs when just the wrong data pattern is transmitted (because that packet winds up failing every time rather then just some of the time). Another possibility in regards to packet loss is simply a bad cable (if you are running GigE over copper). GigE is very sensitive to cable issues and a bad crimp will screw things up easily. In anycase, my feeling about GigE in general is that it's a great cheap way to aggregate data on an uplink, or to an NFS server that needs it, but the incremental cost and hassle factor are still too high to extend the benefit to generic servers. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-stable in the body of the message
NFS problems in 4.5-PRERELEASE
Hi folks, I will do some more research on this problem myself tomorrow (it's fairly late here), but I thought I might already post a message about this today: From what I have read, the fixes to the NFS problems that Jordan mentioned at the beginning of last week are by now in RELENG_4. However, a updated system (world, kernel) as of this morning causes me serious headache. There seems to be an NFS problem, but it's highly strange in nature, and I have not yet been able to figure it out. Anyway, here's what happens: I copied /usr/src, /usr/obj and /usr/ports from one machine to another via NFS. src and obj are fairly normal, ports included about 300 MB worth of distfiles besides the ports skeleton itself. Now, the NFS copy of src and ports does *always* work (I tried 10 times), while copying ports seems to crash my NFS client in 2 out of three cases. The exact panic message I get (which is probably not that useful by itself, but I'll post it anyway) is this: Fatal Trap 12: Page fault while in kernel mode fault virtual address = 0xe8 fault code = supervisor read, page not present instruction pointer = 0x8:0xc01b61cc stack pointer = 0x10:0xe1c73cb8 frame pointer = 0x10:0xe1c73cd4 code segment= base 0x0, limit 0xf, type 0x1b DPL 0, pres 1, def32 1, gran 1 processor eflags= interrupt enabled, resume, IOPL = 0 current process = 367 (cp) interrupt mask = none trap number = 12 panic: page fault And now it gets interesting: This crash does *not* occur randomly, but as my tests have shown, it occurs *always* when /usr/ports/japanese/mnews is being copied (as shown by cp -v). To prevent misunderstandings, let me reword this: The problem seems to occur 2 out of 3 times, but *when* it occurs, then always at japanese/mnews, i.e. always when copying the same file. This fact is it that makes me believe that it's actually a bug in the code and not some hardware / random problem with my machine (if it were, it surely wouldn't occur just at the same file every time...) If anyone has any suggestions as far as this problem is concerned, I'd be glad to hear about it. I'll do some research on this tomorrow and post any interesting results. I may want to set my system up to take crash dump, but before I do that, I'll probably try to find out if there's a pattern somewhere, i.e. if I can crash my machine by *just* copying japanese/mnews, or if I can successfully do several copy attempts of /usr/ports when leaving japanese/mnews out... Good night Nils -- Nils Holland Ti Systems - FreeBSD in Tiddische, Germany http://www.tisys.org * [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-stable in the body of the message
ADDITION: NFS problems in 4.5-PRERELEASE
I actually wanted to go to bed after posting my 4.5-PRE NFS problems message, however, before I actually did that I ran a fsck on the NFS server from which the filesystem causing the problems gets exported. The fsck reported some inconsistencies (though the fs was not exactly marked as dirty). I guess this may actually have had something to do with the problem I reported. I'll let fsck repair the problems and see if I can then successfully do my NFS copy. That will, however, happen tomorrow - I don't think that I can do anything useful at 1:00 am. Good night (this time really!) Nils -- Nils Holland Ti Systems - FreeBSD in Tiddische, Germany http://www.tisys.org * [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-stable in the body of the message
Re: NFS Problems
Alfred Perlstein wrote: * Pat Wendorf [EMAIL PROTECTED] [000925 10:13] wrote: Hello All, I've been using NFS on stable (3.x, 4.x) for a while now, and when it works, it seems to be fast and reliable. However, I've noticed that in any case where it doesn't work (nfs server goes down, cable gets disconnected, weird network card driver issues, etc), the client machines seem to get stuck in an nfs-read state limbo of some sort. My poor laptop was in this state for about 3 days and the daily run's (find in particular) was hung (new hung "find" every day). If I could kill these processes off, I probably wouldn't mind, but these processes simply do not die (even kill -9), without a reboot. The worst part is, while they are hung, I cannot unmount the NFS share (says it's being used). The shutdown isn't clean either because it cannot kill them at shutdown time. Is this normal behavior, and if so, how do I stop it from happening? If this is something I've misconfigured I can provide hardware and config file details on request. Thanks for any help :) You probably want to mount your NFS with the 'intr' and 'soft' options. see the mount_nfs manpage. Thanks Alfred, this did clear up the problem with the machine hanging, and now I believe I'm getting some interesting log messages that may clear up why I've been experiencing this more often than normal: Sep 26 02:28:17 laptop /kernel: nfs_getpages: error 4 Sep 26 02:28:17 laptop /kernel: vm_fault: pager read error, pid 1047 (cp) Sep 26 02:28:52 laptop /kernel: nfs_getpages: error 4 Sep 26 02:28:52 laptop /kernel: vm_fault: pager read error, pid 1053 (cp) Sep 26 02:35:34 laptop /kernel: nfs_getpages: error 4 Sep 26 02:35:34 laptop /kernel: vm_fault: pager read error, pid 1100 (cp) Sep 26 02:36:28 laptop /kernel: nfs_getpages: error 4 Sep 26 02:36:28 laptop /kernel: vm_fault: pager read error, pid 1112 (cp) Sep 26 02:37:04 laptop /kernel: nfs_getpages: error 4 Sep 26 02:37:04 laptop /kernel: vm_fault: pager read error, pid 1118 (cp) My syslog on my laptop keeps repeating this every time I try to do any major I/O on the NFS mount (but thankfully, now I can break out of the cp command when it hangs). The server end doesn't seem to be complaining, just the client end. Thanks again for any help :) -- Pat Wendorf To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-stable" in the body of the message
NFS Problems
Hello All, I've been using NFS on stable (3.x, 4.x) for a while now, and when it works, it seems to be fast and reliable. However, I've noticed that in any case where it doesn't work (nfs server goes down, cable gets disconnected, weird network card driver issues, etc), the client machines seem to get stuck in an nfs-read state limbo of some sort. My poor laptop was in this state for about 3 days and the daily run's (find in particular) was hung (new hung "find" every day). If I could kill these processes off, I probably wouldn't mind, but these processes simply do not die (even kill -9), without a reboot. The worst part is, while they are hung, I cannot unmount the NFS share (says it's being used). The shutdown isn't clean either because it cannot kill them at shutdown time. Is this normal behavior, and if so, how do I stop it from happening? If this is something I've misconfigured I can provide hardware and config file details on request. Thanks for any help :) -- Pat Wendorf To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-stable" in the body of the message
Re: NFS Problems
* Pat Wendorf [EMAIL PROTECTED] [000925 10:13] wrote: Hello All, I've been using NFS on stable (3.x, 4.x) for a while now, and when it works, it seems to be fast and reliable. However, I've noticed that in any case where it doesn't work (nfs server goes down, cable gets disconnected, weird network card driver issues, etc), the client machines seem to get stuck in an nfs-read state limbo of some sort. My poor laptop was in this state for about 3 days and the daily run's (find in particular) was hung (new hung "find" every day). If I could kill these processes off, I probably wouldn't mind, but these processes simply do not die (even kill -9), without a reboot. The worst part is, while they are hung, I cannot unmount the NFS share (says it's being used). The shutdown isn't clean either because it cannot kill them at shutdown time. Is this normal behavior, and if so, how do I stop it from happening? If this is something I've misconfigured I can provide hardware and config file details on request. Thanks for any help :) You probably want to mount your NFS with the 'intr' and 'soft' options. see the mount_nfs manpage. -- -Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]] "I have the heart of a child; I keep it in a jar on my desk." To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-stable" in the body of the message
Re: NFS problems on 4.0-stable.
In message [EMAIL PROTECTED] Jaye Mathisen writes: : got bad cookie vp 0xd24bf1c0 bp 0xc9090500 : got bad cookie vp 0xd24bfda0 bp 0xc906ceb0 Seen these too. Not sure why. Too many other fire to fight. Warner To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-stable" in the body of the message
