Re: how to saturate 100Mbit
On Sun, Dec 14, 2003 at 11:29:07AM +0700, Eugene Grosbein wrote: On Sat, Dec 13, 2003 at 10:17:06AM -0800, Luigi Rizzo wrote: the fxp has a problem which does not allow it to go above 103/110/120kpps depending on which descriptor model you use, no matter how fast the CPU is. Can you explain the problem, please? Sorry if i sound a bit vague but did these tests a couple of years ago, and i do not have access to the fxp specs or contacts with people who designed the chip so i cannot say what is exactly the problem. The problem is in the hardware, not in the driver. Apparently the chip wastes a lot of time (couple of microseconds if i remember well) between reading the descriptor and then transmitting the data. This extra delay is in parallel with some other chip activity, so for larger frames (say 256 bytes) it is completely masked and packets are transmitted at the nominal rate, but smaller packet are sent as if the interframe gap were larger than the standard. At the time i did a lot of experiments to make sure that the NIC was not stalled by the cpu not supplying frames in time, etc., and all tests confirmed the above diagnosis. BTW it is not something that has to do with flow control frames, because enabling/disabling them does not change the throughput (and you can see the frames when they are not disabled). Changing the way descriptor are used (there are two different formats, one has something like the data right after the descriptor) helped increase the performance to 120kpps, whereas the standard freebsd driver is stuck at something between 103 and 110kpps. Not that it matters terribly, just useful to know when you have to push small frames at line rate and you find you can't with those cards. There are others (e.g. intel/dec 2114x, 3com's xl, possibly others, plus probably all of the gbit units when used at 100Mbit) which work fine at line rate with 64-byte frames. Even not using any special kernel modules, a simple loop over a sendto() on a udp socket can achieve around 500kpps on a 2.4GHz box (em or bge). With some tricks and a sufficiently fast PCI bus you can reach some 750kpps but then it really depends on how fast is your PCI bus. A 100*1024*1024/8/1500=8738.1(3) It seems one does not need hundred of thousand pps to achive 100Mbps. certainly not, but because most of the time the overhead is mostly per packet, if you want to run performance tests you care about packets, not data rate. cheers luigi ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: how to saturate 100Mbit
On Sun, Dec 14, 2003 at 11:29:07AM +0700, Eugene Grosbein wrote: 100*1024*1024/8/1500=8738.1(3) SI in bits across a network is base 10, not 2 (1000 vs 1024). -- Richard A Steenbergen [EMAIL PROTECTED] http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC) ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: how to saturate 100Mbit
On Sat, Dec 13, 2003 at 01:06:19AM -0500, Clark Gaylord wrote: Is it possible to saturate 100Mbit ethernet using FreeBSD 4.9-STABLE, Pentium-133 Intel 430VX-based motherboard (PCI-33), Intel 82559 Pro/100 Ethernet (fxp) ? I tried to use sendfile(2) on /dev/zero but that does not work. Then I created 8Tb holey file and used sendfile() on it. That gave 100% CPU load and only 37Mbit/s on wire. I usually use ttcp for tcp throughput measurement. You may not be able to do 100Mbps, due to tcp-ness, but it has a udp mode as well. Depends on what you are interested in doing. But ttcp is pretty low impact. Also, when you use it, boost the buffersize; this can help performance (and lower cpu hit) considerably. Will ttcp or iperf use zero-copy mechanics like sendfile(2) or they will make CPU load by copying from/to kernel space? I made another attempt: set sendspace/recvspace for TCP/UDP to 65536, mounted filesystem using NFS over UDP with -r 65536, -w 65536 and ran from NFS client: dd if=holey-file of=/dev/null bs=10m I've got about 30% of CPU load for the server (P-133) and less than 35mbit/s on wire. Eugene Grosbein ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: how to saturate 100Mbit
dd if=holey-file of=/dev/null bs=10m I've got about 30% of CPU load for the server (P-133) and less than 35mbit/s on wire. Also you can try to dump traffic with tcpdump and send it with /usr/ports/net/tcpreplay I'm trying to send 100Mbit/s for 5-6 minutes with Ethernet frame size at 64 bytes, but I need very good hardware to make this. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: how to saturate 100Mbit
From: DrumFire [mailto:[EMAIL PROTECTED] dd if=holey-file of=/dev/null bs=10m I've got about 30% of CPU load for the server (P-133) and less than 35mbit/s on wire. Also you can try to dump traffic with tcpdump and send it with /usr/ports/net/tcpreplay I'm trying to send 100Mbit/s for 5-6 minutes with Ethernet frame size at 64 bytes, but I need very good hardware to make this. There is a netgraph module called ng_source which can do this. It can achieve about 400Kpps or 1Gbps on a xeon system with a gigabit card, should be able to saturate a fxp. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: how to saturate 100Mbit
On Sat, Dec 13, 2003 at 10:50:33AM -0500, Don Bowman wrote: ... There is a netgraph module called ng_source which can do this. It can achieve about 400Kpps or 1Gbps on a xeon system with a gigabit card, should be able to saturate a fxp. the fxp has a problem which does not allow it to go above 103/110/120kpps depending on which descriptor model you use, no matter how fast the CPU is. Even not using any special kernel modules, a simple loop over a sendto() on a udp socket can achieve around 500kpps on a 2.4GHz box (em or bge). With some tricks and a sufficiently fast PCI bus you can reach some 750kpps but then it really depends on how fast is your PCI bus. cheers luigi [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: how to saturate 100Mbit
On Sat, Dec 13, 2003 at 10:17:06AM -0800, Luigi Rizzo wrote: the fxp has a problem which does not allow it to go above 103/110/120kpps depending on which descriptor model you use, no matter how fast the CPU is. Can you explain the problem, please? Even not using any special kernel modules, a simple loop over a sendto() on a udp socket can achieve around 500kpps on a 2.4GHz box (em or bge). With some tricks and a sufficiently fast PCI bus you can reach some 750kpps but then it really depends on how fast is your PCI bus. 100*1024*1024/8/1500=8738.1(3) It seems one does not need hundred of thousand pps to achive 100Mbps. EUgene ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: how to saturate 100Mbit
Eugene Grosbein wrote: Is it possible to saturate 100Mbit ethernet using FreeBSD 4.9-STABLE, Pentium-133 Intel 430VX-based motherboard (PCI-33), Intel 82559 Pro/100 Ethernet (fxp) ? I tried to use sendfile(2) on /dev/zero but that does not work. Then I created 8Tb holey file and used sendfile() on it. That gave 100% CPU load and only 37Mbit/s on wire. I usually use ttcp for tcp throughput measurement. You may not be able to do 100Mbps, due to tcp-ness, but it has a udp mode as well. Depends on what you are interested in doing. But ttcp is pretty low impact. Also, when you use it, boost the buffersize; this can help performance (and lower cpu hit) considerably. I am doing 800+Mbps on gigE with FreeBSD 5.1 these days. The other trick we use for link utilization with tcp is to multiplex several sessions; we do a few score (or a few hundred :-) simultaneous ttcp's just with a shell script and background the processes. ping -f with larger packets can also be useful. --ckg ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: how to saturate 100Mbit
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 You can also use a tool like iperf to test things like this. Just a thought. nb On Dec 13, 2003, at 12:06 AM, Clark Gaylord wrote: Eugene Grosbein wrote: Is it possible to saturate 100Mbit ethernet using FreeBSD 4.9-STABLE, Pentium-133 Intel 430VX-based motherboard (PCI-33), Intel 82559 Pro/100 Ethernet (fxp) ? I tried to use sendfile(2) on /dev/zero but that does not work. Then I created 8Tb holey file and used sendfile() on it. That gave 100% CPU load and only 37Mbit/s on wire. I usually use ttcp for tcp throughput measurement. You may not be able to do 100Mbps, due to tcp-ness, but it has a udp mode as well. Depends on what you are interested in doing. But ttcp is pretty low impact. Also, when you use it, boost the buffersize; this can help performance (and lower cpu hit) considerably. I am doing 800+Mbps on gigE with FreeBSD 5.1 these days. The other trick we use for link utilization with tcp is to multiplex several sessions; we do a few score (or a few hundred :-) simultaneous ttcp's just with a shell script and background the processes. ping -f with larger packets can also be useful. --ckg ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to [EMAIL PROTECTED] -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.3 (Darwin) iD8DBQE/2q4LFOm2Sy5bRPQRAmO6AJ9tdPc6X8jf6lE0qRqGsjyXnQRZ/QCcDdNw hakmV6u54zdrp7XRyEaMn4M= =05mX -END PGP SIGNATURE- ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to [EMAIL PROTECTED]
