Re: NFS UDP mounts on RELENG_6?
On Mon, 2005-Dec-19 08:39:47 +0100, Oliver Brandmueller wrote: While NFS stalls at the same time ntp to the same host works without problems. So it's not a comüplete stall of all UDP traffic.I guess there's something that's only triggered by a certain combination of things. How about big/fragmented UDP packets? NFS typically sends 8K packets which are split into 6 UDP packets. -- Peter Jeremy ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: NFS UDP mounts on RELENG_6?
A very critical question here is the network topology. UDP NFS _cannot_ be used across switches where the ports are operating at different speeds--unless the UDP packet size is to be smaller than MTU. Be sure and verify that every link between the server and the client are operating at the same speed. -Jon On Sun, 18 Dec 2005, Fabian Keil wrote: Oliver Brandmueller [EMAIL PROTECTED] wrote: On Fri, Dec 16, 2005 at 04:30:31PM +0100, Fabian Keil wrote: Oliver Brandmueller [EMAIL PROTECTED] wrote: I'm experiencing problems when trying to mount NFS filesystems from a RELENG_6 server (FreeBSD hudson 6.0-STABLE FreeBSD 6.0-STABLE #0: Wed Dec 14 16:59:55 CET 2005 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/NFS-32-FBSD6 i386) to either 5.4-STABLE or 6-STABLE clients. mounting works fine, but afterwards the access to the filesystem on the client stalls. As soon as I mount the FS with a TCP mount everything works as expected. The mounts worked fine on UDP when the server was 5.4-STABLE. There is just a plain GigE switch involved, no firewalls or routing. Anyone else experiencing those problems or having an idea? I just copied some files (200 MB) from a NFS Server running FreeBSD africanqueen.local 6.0-STABLE FreeBSD 6.0-STABLE #5: Thu Dec 15 19:31:12 CET 2005 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/AFRICANQUEEN i386 without problems. My client runs FreeBSD 5.4, I use GigE as well, but no switch. Which kind GigE Interface do you use? Client: [EMAIL PROTECTED] ~ $pciconf -lv| grep em0 -A 2 [EMAIL PROTECTED]:1:0: class=0x02 card=0x05491014 chip=0x101e8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82540EP Gigabit Ethernet Controller (Mobile)' Server: [EMAIL PROTECTED] ~ $pciconf -lv| grep re[01] -A 2 [EMAIL PROTECTED]:9:0: class=0x02 card=0x816910ec chip=0x816910ec rev=0x10 hdr=0x00 vendor = 'Realtek Semiconductor' device = 'RTL8169 Gigabit Ethernet Adapter' -- [EMAIL PROTECTED]:10:0: class=0x02 card=0x601b182d chip=0x816910ec rev=0x10 hdr=0x00 vendor = 'Realtek Semiconductor' device = 'RTL8169 Gigabit Ethernet Adapter' re0 is made by Vivanco, re1 is a Sitecom card. Fabian -- http://www.fabiankeil.de/ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
[SOLVED] Re: NFS UDP mounts on RELENG_6?
Hi. On Mon, Dec 19, 2005 at 01:01:34AM -0800, Jon Dama wrote: A very critical question here is the network topology. UDP NFS _cannot_ be used across switches where the ports are operating at different speeds--unless the UDP packet size is to be smaller than MTU. Be sure and verify that every link between the server and the client are operating at the same speed. *ouch* shame on me. I looked at interfaces, links, errors - everything. I found the problem in a misconfiguration and you just pointed at it: The server has not been booted for a few hundred days before upgrading. From an old test there was an mtu 9000 for the NFS interface still in /etc/rc.conf (while it has been reset after failed tests with other hardware on that network manually to 1500). So: SORRY for making everybody mad here. It was just me being blind. Thanx for pointing me at that! - Oliver -- | Oliver Brandmueller | Offenbacher Str. 1 | Germany D-14197 Berlin | | Fon +49-172-3130856 | Fax +49-172-3145027 | WWW: http://the.addict.de/ | | Ich bin das Internet. Sowahr ich Gott helfe. | | Eine gewerbliche Nutzung aller enthaltenen Adressen ist nicht gestattet! | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [SOLVED] Re: NFS UDP mounts on RELENG_6?
No shame. This is a common problem, and if you think real hard you should start getting quasy about UDP NFS under high load. The problem arises because if part of a UDP packet is lost the entire packet is lost, and as UDP NFS uses fixed packet sizes... this means the system never recovers if frames stop droping perpetually. Its easy to see how this happens for speed mismatches, the switch (vs a router) provides no buffering and therefore frames from the fast source are dropped. Now imagine that you have many clients all writing data to the NFS server at once... Suddently that pipe into the server is like a narrow straw. Ooops. I haven't see any evidence that suggests using NFS with UDP is actually useful. IMO, its a false economy. TCP processing takes on the order of 1uS of CPU time--which is on the order of the frame latency through a single switch! That is to say, nothing, but the behavior of TCP NFS under load (when it counts) is superior. TCP SACK and interrupt aggregation are better ways of squeezing extra performance out of your hardware than simply using UDP... Just my two cents. -Jon On Mon, 19 Dec 2005, Oliver Brandmueller wrote: Hi. On Mon, Dec 19, 2005 at 01:01:34AM -0800, Jon Dama wrote: A very critical question here is the network topology. UDP NFS _cannot_ be used across switches where the ports are operating at different speeds--unless the UDP packet size is to be smaller than MTU. Be sure and verify that every link between the server and the client are operating at the same speed. *ouch* shame on me. I looked at interfaces, links, errors - everything. I found the problem in a misconfiguration and you just pointed at it: The server has not been booted for a few hundred days before upgrading. From an old test there was an mtu 9000 for the NFS interface still in /etc/rc.conf (while it has been reset after failed tests with other hardware on that network manually to 1500). So: SORRY for making everybody mad here. It was just me being blind. Thanx for pointing me at that! - Oliver -- | Oliver Brandmueller | Offenbacher Str. 1 | Germany D-14197 Berlin | | Fon +49-172-3130856 | Fax +49-172-3145027 | WWW: http://the.addict.de/ | | Ich bin das Internet. Sowahr ich Gott helfe. | | Eine gewerbliche Nutzung aller enthaltenen Adressen ist nicht gestattet! | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [SOLVED] Re: NFS UDP mounts on RELENG_6?
On Mon, 2005-Dec-19 01:37:44 -0800, Jon Dama wrote: I haven't see any evidence that suggests using NFS with UDP is actually useful. IMO, its a false economy. On modern hardware anyway. Keep in mind that NFS was written to run on a 25MHz (or so) 68020. -- Peter Jeremy ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [SOLVED] Re: NFS UDP mounts on RELENG_6?
I haven't see any evidence that suggests using NFS with UDP is actually useful. IMO, its a false economy. On modern hardware anyway. Keep in mind that NFS was written to run on a 25MHz (or so) 68020. 100% agreed. That is precisely what assumed when I made my statement. -Jon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [SOLVED] Re: NFS UDP mounts on RELENG_6?
On Mon, Dec 19, 2005 at 09:04:46PM +1100, Peter Jeremy wrote.. On Mon, 2005-Dec-19 01:37:44 -0800, Jon Dama wrote: I haven't see any evidence that suggests using NFS with UDP is actually useful. IMO, its a false economy. On modern hardware anyway. Keep in mind that NFS was written to run on a 25MHz (or so) 68020. Sun2, 68010 ;) -- Wilko Bulte [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: NFS UDP mounts on RELENG_6?
Hi. On Fri, Dec 16, 2005 at 04:30:31PM +0100, Fabian Keil wrote: Oliver Brandmueller [EMAIL PROTECTED] wrote: I'm experiencing problems when trying to mount NFS filesystems from a RELENG_6 server (FreeBSD hudson 6.0-STABLE FreeBSD 6.0-STABLE #0: Wed Dec 14 16:59:55 CET 2005 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/NFS-32-FBSD6 i386) to either 5.4-STABLE or 6-STABLE clients. mounting works fine, but afterwards the access to the filesystem on the client stalls. As soon as I mount the FS with a TCP mount everything works as expected. The mounts worked fine on UDP when the server was 5.4-STABLE. There is just a plain GigE switch involved, no firewalls or routing. Anyone else experiencing those problems or having an idea? I just copied some files (200 MB) from a NFS Server running FreeBSD africanqueen.local 6.0-STABLE FreeBSD 6.0-STABLE #5: Thu Dec 15 19:31:12 CET 2005 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/AFRICANQUEEN i386 without problems. My client runs FreeBSD 5.4, I use GigE as well, but no switch. Which kind GigE Interface do you use? I'm currently checking different things. I have another RELENG_6 machine where everything works fine, which has a slightly older system at the moment. Luckily it's still in the testing phase, so I'm currently updating and see if everything is still working after update. This one has bge interfaces (and is amd64). The machine with the problem has em interfaces and I've got the feeling the problem is related to the interface. When doing a tcpdump at the time the mount stalls there's an interesting effect: On the client I see outgoing NFS requests only. The tcpdump on the server shows the incoming requests from the client AND outgoing answers from the server, which don't show up at the client then. I have no other GigE NIC around to put into the original server and it will take some time to do the patching in the datacenter and set up a monitoring port (dunno if the switch is smart enough after all to do that) to sniff from an independent machine. I'll keep you posted. - Oliver -- | Oliver Brandmueller | Offenbacher Str. 1 | Germany D-14197 Berlin | | Fon +49-172-3130856 | Fax +49-172-3145027 | WWW: http://the.addict.de/ | | Ich bin das Internet. Sowahr ich Gott helfe. | | Eine gewerbliche Nutzung aller enthaltenen Adressen ist nicht gestattet! | pgpUbyUO2tENa.pgp Description: PGP signature
Re: NFS UDP mounts on RELENG_6?
Hi. On Sun, Dec 18, 2005 at 01:45:37PM +0100, Oliver Brandmueller wrote: Which kind GigE Interface do you use? I'm currently checking different things. I have another RELENG_6 machine where everything works fine, which has a slightly older system at the moment. Luckily it's still in the testing phase, so I'm currently updating and see if everything is still working after update. This one has bge interfaces (and is amd64). OK. UDP NFS works as expected with amd64/bge on RELENG_6 of today. It does not work in the same network on i386/em for me. Still investigating. - Olli -- | Oliver Brandmueller | Offenbacher Str. 1 | Germany D-14197 Berlin | | Fon +49-172-3130856 | Fax +49-172-3145027 | WWW: http://the.addict.de/ | | Ich bin das Internet. Sowahr ich Gott helfe. | | Eine gewerbliche Nutzung aller enthaltenen Adressen ist nicht gestattet! | pgpButOTw5FWE.pgp Description: PGP signature
Re: NFS UDP mounts on RELENG_6?
On Sunday 18 December 2005 15:12, Oliver Brandmueller wrote: Hi. On Sun, Dec 18, 2005 at 01:45:37PM +0100, Oliver Brandmueller wrote: Which kind GigE Interface do you use? I'm currently checking different things. I have another RELENG_6 machine where everything works fine, which has a slightly older system at the moment. Luckily it's still in the testing phase, so I'm currently updating and see if everything is still working after update. This one has bge interfaces (and is amd64). OK. UDP NFS works as expected with amd64/bge on RELENG_6 of today. It does not work in the same network on i386/em for me. Still investigating. To rule out NFS, do something like syslogging remotely (also UDP). -- Melvyn Sopacua [EMAIL PROTECTED] FreeBSD 6.0-STABLE Qt: 3.3.5 KDE: 3.4.3 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: NFS UDP mounts on RELENG_6?
Oliver Brandmueller [EMAIL PROTECTED] wrote: On Fri, Dec 16, 2005 at 04:30:31PM +0100, Fabian Keil wrote: Oliver Brandmueller [EMAIL PROTECTED] wrote: I'm experiencing problems when trying to mount NFS filesystems from a RELENG_6 server (FreeBSD hudson 6.0-STABLE FreeBSD 6.0-STABLE #0: Wed Dec 14 16:59:55 CET 2005 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/NFS-32-FBSD6 i386) to either 5.4-STABLE or 6-STABLE clients. mounting works fine, but afterwards the access to the filesystem on the client stalls. As soon as I mount the FS with a TCP mount everything works as expected. The mounts worked fine on UDP when the server was 5.4-STABLE. There is just a plain GigE switch involved, no firewalls or routing. Anyone else experiencing those problems or having an idea? I just copied some files (200 MB) from a NFS Server running FreeBSD africanqueen.local 6.0-STABLE FreeBSD 6.0-STABLE #5: Thu Dec 15 19:31:12 CET 2005 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/AFRICANQUEEN i386 without problems. My client runs FreeBSD 5.4, I use GigE as well, but no switch. Which kind GigE Interface do you use? Client: [EMAIL PROTECTED] ~ $pciconf -lv| grep em0 -A 2 [EMAIL PROTECTED]:1:0: class=0x02 card=0x05491014 chip=0x101e8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82540EP Gigabit Ethernet Controller (Mobile)' Server: [EMAIL PROTECTED] ~ $pciconf -lv| grep re[01] -A 2 [EMAIL PROTECTED]:9:0: class=0x02 card=0x816910ec chip=0x816910ec rev=0x10 hdr=0x00 vendor = 'Realtek Semiconductor' device = 'RTL8169 Gigabit Ethernet Adapter' -- [EMAIL PROTECTED]:10:0: class=0x02 card=0x601b182d chip=0x816910ec rev=0x10 hdr=0x00 vendor = 'Realtek Semiconductor' device = 'RTL8169 Gigabit Ethernet Adapter' re0 is made by Vivanco, re1 is a Sitecom card. Fabian -- http://www.fabiankeil.de/ signature.asc Description: PGP signature
Re: NFS UDP mounts on RELENG_6?
Hi Melvyn. On Sun, Dec 18, 2005 at 03:28:21PM +0100, Melvyn Sopacua wrote: OK. UDP NFS works as expected with amd64/bge on RELENG_6 of today. It does not work in the same network on i386/em for me. Still investigating. To rule out NFS, do something like syslogging remotely (also UDP). While NFS stalls at the same time ntp to the same host works without problems. So it's not a comüplete stall of all UDP traffic.I guess there's something that's only triggered by a certain combination of things. At the moment I'm building RELENG_6_0 on the server to find out, if the regression was from 5 to 6 or from 6.0 to 6-STABLE. - Oliver -- | Oliver Brandmueller | Offenbacher Str. 1 | Germany D-14197 Berlin | | Fon +49-172-3130856 | Fax +49-172-3145027 | WWW: http://the.addict.de/ | | Ich bin das Internet. Sowahr ich Gott helfe. | | Eine gewerbliche Nutzung aller enthaltenen Adressen ist nicht gestattet! | pgpKA3JTVoc4P.pgp Description: PGP signature
Re: NFS UDP mounts on RELENG_6?
Hi. On Sun, Dec 18, 2005 at 04:50:29PM +0100, Fabian Keil wrote: Which kind GigE Interface do you use? Client: [EMAIL PROTECTED] ~ $pciconf -lv| grep em0 -A 2 [EMAIL PROTECTED]:1:0: class=0x02 card=0x05491014 chip=0x101e8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82540EP Gigabit Ethernet Controller (Mobile)' Server: [EMAIL PROTECTED] ~ $pciconf -lv| grep re[01] -A 2 [EMAIL PROTECTED]:9:0: class=0x02 card=0x816910ec chip=0x816910ec rev=0x10 hdr=0x00 vendor = 'Realtek Semiconductor' device = 'RTL8169 Gigabit Ethernet Adapter' -- [EMAIL PROTECTED]:10:0: class=0x02 card=0x601b182d chip=0x816910ec rev=0x10 hdr=0x00 vendor = 'Realtek Semiconductor' device = 'RTL8169 Gigabit Ethernet Adapter' re0 is made by Vivanco, re1 is a Sitecom card. So at least the server does not have an em interface in your case. I have an i386 RELENG_6 client running (on a 5.4 server) with an em interface for NFS and don't have any problems with this one. Also my em interface is a different chip: [EMAIL PROTECTED]:2:0: class=0x02 card=0x10768086 chip=0x10768086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82547EI Gigabit Ethernet Controller' class= network subclass = ethernet em1: Intel(R) PRO/1000 Network Connection, Version - 2.1.7 port 0xc400-0xc43f mem 0xf202-0xf203 ,0xf200-0xf201 irq 17 at device 2.0 on pci3 It's actually mentioned in CVS log for src/sys/dev/em/if_em_hw.c in the last commit: --- cut here --- revision 1.89 date: 2005/11/24 01:44:48; author: glebius; state: Exp; lines: +131 -77 Merge in new driver version from Intel - 3.2.18. The most important change is support for adapters based on 82571 and 82572 chips. Tested on: 82547EI on i386 Tested on: 82540EM on sparc64 Everything has been fine on 5.4, though. - Oliver -- | Oliver Brandmueller | Offenbacher Str. 1 | Germany D-14197 Berlin | | Fon +49-172-3130856 | Fax +49-172-3145027 | WWW: http://the.addict.de/ | | Ich bin das Internet. Sowahr ich Gott helfe. | | Eine gewerbliche Nutzung aller enthaltenen Adressen ist nicht gestattet! | pgp1702kOLLs9.pgp Description: PGP signature
Re: NFS UDP mounts on RELENG_6?
Oliver Brandmueller [EMAIL PROTECTED] wrote: I'm experiencing problems when trying to mount NFS filesystems from a RELENG_6 server (FreeBSD hudson 6.0-STABLE FreeBSD 6.0-STABLE #0: Wed Dec 14 16:59:55 CET 2005 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/NFS-32-FBSD6 i386) to either 5.4-STABLE or 6-STABLE clients. mounting works fine, but afterwards the access to the filesystem on the client stalls. As soon as I mount the FS with a TCP mount everything works as expected. The mounts worked fine on UDP when the server was 5.4-STABLE. There is just a plain GigE switch involved, no firewalls or routing. Anyone else experiencing those problems or having an idea? I just copied some files (200 MB) from a NFS Server running FreeBSD africanqueen.local 6.0-STABLE FreeBSD 6.0-STABLE #5: Thu Dec 15 19:31:12 CET 2005 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/AFRICANQUEEN i386 without problems. My client runs FreeBSD 5.4, I use GigE as well, but no switch. Fabian -- http://www.fabiankeil.de/ signature.asc Description: PGP signature
