Re: Proposed 6.2 em RELEASE patch
Hi All, Unfortunately our company hasn't had the resources to help FreeBSD much over the years, but I do want to say thank you to the folks who are helping sort out this issue with the em driver. That Intel gigabit interface is very, very common on server hardware nowadays and it means a lot to us to make sure it (and FreeBSD 6.2 in general) are stable. - mike Michael F. DeMan Director of Technology OpenAccess Network Services Bellingham, WA 98225 [EMAIL PROTECTED] 360-647-0785 --- If your question is support related, please contact [EMAIL PROTECTED] directly for faster assistance --- On Nov 13, 2006, at 3:15 PM, Мирослав Славков wrote: Hey. I've got one new machine for testing for 1-2 days... here's some output.. With the latest drivers (cvsup'ed from yesterday) Send box: 2x Intel Xeon 5110 (1.6GHz), SuperMicro X7-DBE, Intel Pro/ 1000 MT Server Adapter CPU: Intel(R) Xeon(R) CPU5110 @ 1.60GHz (1600.01-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6 Features=0xbfebfbffGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE > Features2=0x4e33d,CX16,,, > AMD Features=0x2010 AMD Features2=0x1 Cores per package: 2 real memory = 1073086464 (1023 MB) avail memory = 1040961536 (992 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs Recv box: AMD Sempron 3800+, (Some nVidia chipset.. maybe 4), Intel Pro/1000 PT Desktop Adapter system load is around 2 EM_INTR_FAST is defined (if not defined load is higher..5-6) [EMAIL PROTECTED] ~]# iperf -c 172.16.133.2 -t 999 Client connecting to 172.16.133.2, TCP port 5001 TCP window size: 129 KByte (default) [ 3] local 172.16.133.1 port 58425 connected with 172.16.133.2 port 5001 ^C[ 3] 0.0-48.1 sec 4.87 GBytes869 Mbits/sec [EMAIL PROTECTED] ~]# ifstat -b em0 em2 Kbps in Kbps out Kbps in Kbps out 46117.87 885607.5 12.76 1.45 45892.97 879815.3 22.61 1.07 ifstat: warning: rollover for interface em0, reinitialising. 0.00 0.00581.95 1.07 46436.31 890665.0700.36 2.27 46062.60 884031.1794.26 1.07 46576.76 893408.4846.86 1.07 47117.81 903658.6859.40 1.07 46141.71 885471.3867.05 1.07 45926.92 881372.1363.41 1.07 46003.51 883432.4 35.64 1.07 46052.45 884164.9 28.41 1.07 45863.03 880448.1 12.66 1.07 46152.73 885549.1 33.71 1.07 46737.09 896811.9 39.02 1.07 46188.92 885847.3 58.92 1.07 45770.39 878602.0 19.24 1.07 46743.62 896967.0 34.72 1.07 46412.27 890970.9 34.65 1.07 46322.73 888747.3 26.61 1.07 46002.64 882522.8 23.69 1.07 46450.56 891662.8 26.91 1.07 45991.55 882605.9 37.92 1.07 46067.28 883815.5 34.82 1.07 46120.28 885357.6 33.79 1.07 46133.17 885316.5 21.56 1.07 46700.43 895554.6 10.12 1.49 45476.68 873874.2 15.78 1.07 45791.88 878655.9 15.99 1.07 46344.84 889292.8 7.72 1.07 46577.31 893572.1 10.49 1.07 46065.09 884277.6 8.77 1.07 46566.97 892898.4 6.07 1.07 46232.46 886846.8 13.02 1.07 46165.29 886647.9 7.47 1.07 46080.80 884256.7 14.54 1.07 46757.57 896977.8 14.59 1.07 46182.78 885594.3 10.30 1.07 46103.81 885661.0 13.95 1.07 46469.89 891511.3 13.85 1.07 46470.86 891611.1 11.42 1.07 ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
Mike Tancsa wrote: At 12:50 PM 11/13/2006, Ivan Voras wrote: Mike Tancsa wrote: > At 12:15 AM 11/13/2006, Scott Long wrote: > >> Is this with EM_INTR_FAST enabled also? > > Yes. Havent done the stock case yet, but will do so later today. Do you have a comparison with Linux under the same circumstances? I had a disk with 64bit already installed. I will try with 32bit tomorrow. I can also try FreeBSD AMD64 on the box to see how it does. ifstat gives a bit of an odd output, but its the same sort of pattern where adding a second stream in the same direction, slows down the first one. On the box R2 ... The box is totally responsive throughout with no packet loss on the management interface However, it seems quite a bit slower than FreeBSD when its tweaked with ADAPTIVE_GIANT removed... But again, this is 64bit so not quite apples to apples yet. Also, I need to check the default driver config to see if their NAPI or whatever its called is enabled. More tests to come. ---Mike More excellent data, thanks. I have some changes on the drawing board that should significantly improve forwarding and bridging in the em driver. Do you have a limit on how much more time you can spend on these tests? It might be a week or more before I have anything that can be tested. Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
Hey. I've got one new machine for testing for 1-2 days... here's some output.. With the latest drivers (cvsup'ed from yesterday) Send box: 2x Intel Xeon 5110 (1.6GHz), SuperMicro X7-DBE, Intel Pro/1000 MT Server Adapter CPU: Intel(R) Xeon(R) CPU5110 @ 1.60GHz (1600.01-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6 Features=0xbfebfbff Features2=0x4e33d,CX16,,,> AMD Features=0x2010 AMD Features2=0x1 Cores per package: 2 real memory = 1073086464 (1023 MB) avail memory = 1040961536 (992 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs Recv box: AMD Sempron 3800+, (Some nVidia chipset.. maybe 4), Intel Pro/1000 PT Desktop Adapter system load is around 2 EM_INTR_FAST is defined (if not defined load is higher..5-6) [EMAIL PROTECTED] ~]# iperf -c 172.16.133.2 -t 999 Client connecting to 172.16.133.2, TCP port 5001 TCP window size: 129 KByte (default) [ 3] local 172.16.133.1 port 58425 connected with 172.16.133.2 port 5001 ^C[ 3] 0.0-48.1 sec 4.87 GBytes869 Mbits/sec [EMAIL PROTECTED] ~]# ifstat -b em0 em2 Kbps in Kbps out Kbps in Kbps out 46117.87 885607.5 12.76 1.45 45892.97 879815.3 22.61 1.07 ifstat: warning: rollover for interface em0, reinitialising. 0.00 0.00581.95 1.07 46436.31 890665.0700.36 2.27 46062.60 884031.1794.26 1.07 46576.76 893408.4846.86 1.07 47117.81 903658.6859.40 1.07 46141.71 885471.3867.05 1.07 45926.92 881372.1363.41 1.07 46003.51 883432.4 35.64 1.07 46052.45 884164.9 28.41 1.07 45863.03 880448.1 12.66 1.07 46152.73 885549.1 33.71 1.07 46737.09 896811.9 39.02 1.07 46188.92 885847.3 58.92 1.07 45770.39 878602.0 19.24 1.07 46743.62 896967.0 34.72 1.07 46412.27 890970.9 34.65 1.07 46322.73 888747.3 26.61 1.07 46002.64 882522.8 23.69 1.07 46450.56 891662.8 26.91 1.07 45991.55 882605.9 37.92 1.07 46067.28 883815.5 34.82 1.07 46120.28 885357.6 33.79 1.07 46133.17 885316.5 21.56 1.07 46700.43 895554.6 10.12 1.49 45476.68 873874.2 15.78 1.07 45791.88 878655.9 15.99 1.07 46344.84 889292.8 7.72 1.07 46577.31 893572.1 10.49 1.07 46065.09 884277.6 8.77 1.07 46566.97 892898.4 6.07 1.07 46232.46 886846.8 13.02 1.07 46165.29 886647.9 7.47 1.07 46080.80 884256.7 14.54 1.07 46757.57 896977.8 14.59 1.07 46182.78 885594.3 10.30 1.07 46103.81 885661.0 13.95 1.07 46469.89 891511.3 13.85 1.07 46470.86 891611.1 11.42 1.07 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
At 12:50 PM 11/13/2006, Ivan Voras wrote: Mike Tancsa wrote: > At 12:15 AM 11/13/2006, Scott Long wrote: > >> Is this with EM_INTR_FAST enabled also? > > Yes. Havent done the stock case yet, but will do so later today. Do you have a comparison with Linux under the same circumstances? I had a disk with 64bit already installed. I will try with 32bit tomorrow. I can also try FreeBSD AMD64 on the box to see how it does. ifstat gives a bit of an odd output, but its the same sort of pattern where adding a second stream in the same direction, slows down the first one. On the box R2 [EMAIL PROTECTED] ifstat-1.1]# ifstat -b eth0eth1eth3eth4 Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out 0.00 0.00 0.00 0.00 0.00 0.00 4.89 3.74 0.00 0.00 0.00 0.00 0.00 0.00 0.50 1.45 0.00 0.00 0.00 0.00 0.00 0.00 1.00 1.45 160965.0 0.00 0.00 0.00 0.00 0.00 0.83 1.95 0.00 0.00 0.00 272056.4 0.00 0.00 1.00 1.45 393994.2 0.00 0.00 0.00 0.00 0.00 5.47 1.45 0.00 0.00 0.00 393543.7 0.00 0.00 4.25 1.45 392911.0 0.00 0.00 0.00 0.00 0.00 2.50 1.45 0.00 0.00 0.50 392756.4 0.00 0.00 1.25 1.45 392626.7 0.00 0.00 0.00 0.00 0.00 1.75 1.45 0.00 0.00 0.00 393233.9 0.00 0.00 6.44 1.45 424068.1 0.00 0.00 0.00 0.00 0.00 1.74 1.45** 0.00 0.00 0.00 460503.1 0.00 0.00 2.72 1.45 509218.1 0.00 0.00 0.00 0.00 0.00 0.99 1.45 0.00 0.00 0.00 507800.4 0.00 0.00 0.50 1.45 502649.5 0.00 0.00 0.00 0.00 0.00 1.00 1.45 0.00 0.00 0.50 507537.1 0.00 0.00 0.50 1.46 519717.9 0.00 0.00 0.00 0.00 0.00 1.00 1.45 0.00 0.00 0.00 525973.4 0.00 0.00 0.50 1.46 520609.0 0.00 0.00 0.00 0.00 0.00 1.00 1.45 0.00 0.00 0.00 517888.6 0.00 0.00 0.50 1.45 525957.3 0.00 0.00 0.00 0.00 0.00 1.00 1.46 0.00 0.00 0.00 524119.9 0.00 0.00 0.50 1.45 522671.1 0.00 0.00 0.00 0.00 0.00 0.99 1.44 0.00 0.00 0.00 494008.7 0.00 0.00 0.50 1.45 390666.3 0.00 0.00 0.00 0.00 0.00 1.00 1.45 0.00 0.00 0.00 273779.6 0.00 0.00 0.50 1.45 0.00 0.00 0.00 0.00 0.00 0.00 1.00 1.45 0.00 0.00 0.00 0.00 0.00 0.00 0.50 1.45 [EMAIL PROTECTED] ifstat-1.1]# I added the second stream, going in the same direction at ** On one of the targets running netreceive you can see the impact. [tyan-1u]# ifstat -b rl0 bge0 Kbps in Kbps out Kbps in Kbps out 0.94 1.42 182716.2 0.00 0.47 1.05 182299.5 0.00 0.94 1.05 182493.4 0.33 0.94 2.09 182588.7 0.00 0.94 1.05 181959.8 0.00 0.47 1.05 104949.7 0.00 0.94 1.05 95674.27 0.00 0.47 1.05 95930.79 0.00 0.94 1.05 98329.93 0.00 0.94 1.05 97940.21 0.00 0.94 1.05 100636.9 0.00 0.47 1.05 99879.34 0.00 ^C [tyan-1u]# When the packets are bi-directional, the impact is not as great in LINUX as it is on FreeBSD [EMAIL PROTECTED] ifstat-1.1]# ifstat -b eth0eth1eth3eth4 Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out 0.00 0.00 0.00 0.00 0.00 0.00 3.65 10.81 0.00 0.00 0.00 0.00 0.00 0.00 0.50 1.45 0.00 0.00 0.00 0.00 0.00 0.00 0.83 1.95 0.00 0.00 0.00 0.00 0.00 0.00 1.50 8.03 0.00 0.00 0.00 0.00 0.00 0.00 0.50 1.45 0.00 0.00 0.00 0.00 0.00 0.00 1.00 1.45 0.00 230009.2 0.00 0.00 0.00 0.00 2.83 51.22 0.00 0.00 334969.3 0.00 0.00 0.00 1.00 1.45 0.00 369184.5 0.00 0.00 0.00 0.00 0.50 1.45 0.00 0.00 369294.2 0.00 0.00 0.00 3.33 51.10 0.00 367348.7 0.00 0.00 0.00 0.00 0.50 1.45 0.00 0.00 3
Re: Proposed 6.2 em RELEASE patch
Mike Tancsa wrote: At 12:15 AM 11/13/2006, Scott Long wrote: Is this with EM_INTR_FAST enabled also? Without it, the 2 streams are definitely lossy on the management interface ---Mike Ok, and would you be able to test the polling options as well? Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
At 12:15 AM 11/13/2006, Scott Long wrote: Is this with EM_INTR_FAST enabled also? Without it, the 2 streams are definitely lossy on the management interface ---Mike ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
Mike Tancsa wrote: > At 12:15 AM 11/13/2006, Scott Long wrote: > >> Is this with EM_INTR_FAST enabled also? > > Yes. Havent done the stock case yet, but will do so later today. Do you have a comparison with Linux under the same circumstances? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
At 12:15 AM 11/13/2006, Scott Long wrote: Is this with EM_INTR_FAST enabled also? Yes. Havent done the stock case yet, but will do so later today. ---Mike ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
Mike Tancsa wrote: At 11:05 PM 11/12/2006, Scott Long wrote: Mike Tancsa wrote: However, if I turn on fastforwarding, its back to the old behavior with it locking up. This was with the stock driver. I will try the same test with #define EM_FAST_INTR 1 as well as taking out the nfs option from the kernel driver. Anything else to tune with ? ---Mike Would you mind going so far as to adding NO_ADAPTIVE_MUTEXES? This is Wow, it totally survives 2 streams (2 in one direction or 2 in opposite directions)... AND it almost can survive a 3rd! It does seem to be dropping packets on em1 when I add in a second stream blasting in the same direction, but its survives. You can see at ** I add in the second stream, and the outbound on em1 drops, even though more packets are going through. I stop the second stream then add a stream going in the other direction. Oddly enough, with a UP kernel it cant do 2 streams, just the one. I thought the extra locking on SMP would make it worse ? In this case it seems no. Is this with EM_INTR_FAST enabled also? Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
At 11:05 PM 11/12/2006, Scott Long wrote: Mike Tancsa wrote: However, if I turn on fastforwarding, its back to the old behavior with it locking up. This was with the stock driver. I will try the same test with #define EM_FAST_INTR 1 as well as taking out the nfs option from the kernel driver. Anything else to tune with ? ---Mike Would you mind going so far as to adding NO_ADAPTIVE_MUTEXES? This is Wow, it totally survives 2 streams (2 in one direction or 2 in opposite directions)... AND it almost can survive a 3rd! It does seem to be dropping packets on em1 when I add in a second stream blasting in the same direction, but its survives. You can see at ** I add in the second stream, and the outbound on em1 drops, even though more packets are going through. I stop the second stream then add a stream going in the other direction. Oddly enough, with a UP kernel it cant do 2 streams, just the one. I thought the extra locking on SMP would make it worse ? In this case it seems no. em0 em1 bge1 Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out 0.00 0.00 0.00 0.00 0.47 1.17 0.00 0.00 0.00 0.00 0.47 1.17 0.00 0.00 0.00 0.00 0.94 1.17 184837.3 0.00 0.00 184634.5 0.47 1.17 416856.2 0.00 0.00 416232.3 0.94 1.17 414080.2 0.00 0.00 413829.7 0.47 1.17 415656.3 0.00 0.00 415449.1 1.40 1.17 413015.0 0.00 0.00 412519.6 0.94 1.17 415094.8 0.00 0.00 414795.6 1.87 1.17 415548.6 0.00 0.00 415323.7 0.47 1.17 416629.6 0.00 0.00 416046.7 1.40 1.17 417032.5 0.00 0.00 416854.1 0.47 1.17 416606.9 0.00 0.00 416217.9 0.94 1.17 591084.4 0.00 0.00 307308.9 0.47 1.17** 692232.6 0.00 0.00 248915.3 1.41 2.34 692837.6 0.00 0.00 248765.0 0.47 1.17 690216.7 0.00 0.00 245400.2 1.26 1.59 697517.9 0.00 0.00 245689.7 0.94 2.34 697039.0 0.00 0.00 245553.8 1.40 3.09 697424.1 0.00 0.00 244360.9 0.47 1.17 692166.5 0.00 0.00 245316.4 0.94 1.17 697102.3 0.00 0.00 244969.0 0.47 1.17 560005.0 0.00 0.00 329704.8 0.94 3.09 405189.2 0.00 0.00 404969.8 0.94 1.17 417511.0 0.00 0.00 416902.5 0.47 1.17 415029.0 0.00 0.00 414745.3 0.94 1.17 411610.7 15798.46 49760.74 347870.8 0.94 1.17 417520.1 55251.77 174393.0 187379.1 1.40 2.34 414477.5 56203.05 174188.7 185302.4 0.47 1.17 em0 em1 bge1 Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out 416985.7 55063.17 174285.5 186733.5 0.94 2.34 415291.6 55358.65 174493.7 184466.1 1.41 5.01 409410.6 57657.47 174494.3 186945.9 0.94 1.17 417197.4 55271.79 173405.6 189667.3 0.94 3.09 415310.8 55503.47 174283.4 186058.0 0.94 1.17 417290.4 55983.99 174279.4 185274.7 0.94 4.59 418032.6 55648.61 174424.4 186057.0 0.94 1.17 416777.8 55283.73 174886.1 185034.6 0.47 1.17 414412.4 33989.34 105397.4 276853.1 0.94 1.17 416477.8 0.00 0.00 415984.6 0.47 1.17 416656.7 0.00 0.00 416117.0 0.94 1.17 416527.4 0.00 0.00 416372.2 0.47 1.17 250260.8 0.00 0.00 250029.2 0.94 1.17 0.00 0.00 0.00 0.00 0.47 1.17 0.00 0.00 0.00 0.00 0.94 1.17 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
Mike Tancsa wrote: However, if I turn on fastforwarding, its back to the old behavior with it locking up. This was with the stock driver. I will try the same test with #define EM_FAST_INTR 1 as well as taking out the nfs option from the kernel driver. Anything else to tune with ? ---Mike Would you mind going so far as to adding NO_ADAPTIVE_MUTEXES? This is excellent data, thanks a lot for working on it. Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
At 08:47 PM 11/12/2006, Scott Long wrote: 2. Try compiling in WITNESS and running the test as before, then break into the debugger as before. Run 'show locks'. I'm not sure how fruitful this will be, WITNESS might make it unbearably slow. It was in that kernel already So you're seeing the livelock under load while also having WITNESS enabled? Have you tried your test without WITNESS? What about INVARIANTS? Sorry, by it was in the kernel already, it was from last nights tests where you asked me to break into the debugger to see what was going on. I figured you would want it in there. My original tests were all without WITNESS and INVARIANTS and they were showing the same behaviour (locking up on the management interface) with and without WITNESS and INVARIANTS Removing ADAPTIVE_GIANT seems to be very key in this! If I remove it, the box no longer locks up under a high packet rate! Its still a little sluggish on the bge OOB interface, but its not totally locking up like before Running ifstat -b from the management interface, here is the traffic pattern first with one blast running and then the second kicks in. The Kbps/s out on em1 drops in half, but the box is still responsive and bge1... Certainly responsive enough to see the output of ifstat. 218028.4 0.00 0.00 216389.9 0.47 1.17 218971.2 0.00 0.00 217123.1 0.94 2.34 217754.7 0.00 0.00 215677.2 0.94 1.17 215763.1 0.00 0.00 214578.5 0.47 1.17 217850.3 0.00 0.00 216138.7 1.40 2.34 217668.0 0.00 0.00 215888.3 0.47 1.17 em0 em1 bge1 Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out 217382.3 0.00 0.00 216293.8 1.26 1.59 218247.9 118916.5 234167.3 124972.1 0.47 2.17 215973.4 220459.9 393350.0 54875.64 1.40 2.34 217697.7 210984.5 398543.3 58165.20 0.94 3.09 217183.6 200424.9 399096.5 61845.74 0.94 1.17 217788.7 221212.9 400129.8 53716.38 0.47 1.17 217040.0 204934.2 396628.7 61247.34 1.40 2.34 217105.9 201295.5 399274.1 61710.76 0.94 3.09 215605.6 225012.4 397533.5 53237.31 0.94 1.17 217933.3 200346.4 395949.3 62734.52 0.47 1.17 216783.7 205114.1 395923.0 61042.17 0.94 1.17 217885.7 88513.41 159056.4 151418.9 0.47 1.17 216927.2 0.00 0.00 215450.4 0.94 1.17 217454.1 0.00 0.00 215466.6 0.47 1.17 216810.6 0.00 0.00 215564.2 1.40 1.17 217608.3 0.00 0.00 216061.8 0.47 1.17 217452.8 0.00 0.00 215627.1 0.94 1.17 217185.0 0.00 0.00 216079.7 0.47 1.17 However, if I turn on fastforwarding, its back to the old behavior with it locking up. This was with the stock driver. I will try the same test with #define EM_FAST_INTR 1 as well as taking out the nfs option from the kernel driver. Anything else to tune with ? ---Mike Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
Mike Tancsa wrote: At 11:41 AM 11/12/2006, Scott Long wrote: Mike Tancsa wrote: At 01:42 AM 11/11/2006, Scott Long wrote: driver. What will help me is if you can hook up a serial console to your machine and see if it can be made to drop to the debugger while it is under load and otherwise unresponsive. If you can, getting a process dump might help confirm where each CPU is spending its time. ./netblast 192.168.88.218 500 110 1000 I compiled in the various debugging options and on the serial console I get a few Expensive timeout(9) function: 0xc0601e48(0) 0.024135749 s and the serial console One CPU seems to be stuck in softclock. My guess here is that there is significant lock contention. Please try the following: 1. Use addr2line or gdb on the kernel to find out what function is at 0xc0601e48 (the address that was printed above). This address will change every time you recompile the kernel, so you'll need to reacquire it from the console each time. # addr2line 0xc0601e48 -e kernel.debug -f nfsrv_timer /usr/src/sys/nfsserver/nfs_srvsock.c:809 # Can you try removing NFS from your kernel? and # addr2line 0xc0561444 -e kernel.debug -f sleepq_timeout /usr/src/sys/kern/subr_sleepqueue.c:724 This is sched_lock contention. I dont have any nfs activity on the box 2. Try compiling in WITNESS and running the test as before, then break into the debugger as before. Run 'show locks'. I'm not sure how fruitful this will be, WITNESS might make it unbearably slow. It was in that kernel already So you're seeing the livelock under load while also having WITNESS enabled? Have you tried your test without WITNESS? What about INVARIANTS? Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
At 11:41 AM 11/12/2006, Scott Long wrote: Mike Tancsa wrote: At 01:42 AM 11/11/2006, Scott Long wrote: driver. What will help me is if you can hook up a serial console to your machine and see if it can be made to drop to the debugger while it is under load and otherwise unresponsive. If you can, getting a process dump might help confirm where each CPU is spending its time. ./netblast 192.168.88.218 500 110 1000 I compiled in the various debugging options and on the serial console I get a few Expensive timeout(9) function: 0xc0601e48(0) 0.024135749 s and the serial console One CPU seems to be stuck in softclock. My guess here is that there is significant lock contention. Please try the following: 1. Use addr2line or gdb on the kernel to find out what function is at 0xc0601e48 (the address that was printed above). This address will change every time you recompile the kernel, so you'll need to reacquire it from the console each time. # addr2line 0xc0601e48 -e kernel.debug -f nfsrv_timer /usr/src/sys/nfsserver/nfs_srvsock.c:809 # and # addr2line 0xc0561444 -e kernel.debug -f sleepq_timeout /usr/src/sys/kern/subr_sleepqueue.c:724 I dont have any nfs activity on the box 2. Try compiling in WITNESS and running the test as before, then break into the debugger as before. Run 'show locks'. I'm not sure how fruitful this will be, WITNESS might make it unbearably slow. It was in that kernel already All it would generally show is exclusive spin mutex sio r = 0 (0xc07c81c0) locked @ /usr/src/sys/dev/sio/sio.c:1390 But I did notice in /var/log/kernel Nov 12 00:04:09 R2 kernel: exclusive spin mutex sio r = 0 (0xc07c81c0) locked @ /usr/src/sys/dev/sio/sio.c:1390 Nov 12 00:04:09 R2 kernel: exclusive sleep mutex em1 (network driver) r = 0 (0xc64bc9d4) locked @ /usr/src/sys/dev/ em/if_em.c:3569 Nov 12 00:04:09 R2 kernel: exclusive spin mutex sio r = 0 (0xc07c81c0) locked @ /usr/src/sys/dev/sio/sio.c:1390 Nov 12 19:27:39 R2 kernel: KDB: enter: Line break on console Nov 12 19:27:39 R2 kernel: exclusive spin mutex sio r = 0 (0xc07c81c0) locked @ /usr/src/sys/dev/sio/sio.c:1390 Nov 12 19:28:13 R2 kernel: KDB: enter: Line break on console Nov 12 19:28:13 R2 kernel: exclusive spin mutex sio r = 0 (0xc07c81c0) locked @ /usr/src/sys/dev/sio/sio.c:1390 Nov 12 19:31:37 R2 kernel: KDB: enter: Line break on console Nov 12 19:31:37 R2 kernel: exclusive spin mutex sio r = 0 (0xc07c81c0) locked @ /usr/src/sys/dev/sio/sio.c:1390 Nov 12 19:34:22 R2 kernel: KDB: enter: Line break on console Nov 12 19:34:22 R2 kernel: exclusive spin mutex sio r = 0 (0xc07c81c0) locked @ /usr/src/sys/dev/sio/sio.c:1390 3. Turn on MUTEX_PROFILING according to the text in /sys/conf/NOTES and the man page by the same name. 4. Remove ADAPTIVE_GIANT from your config and retest. If that doesn't show a difference, then try setting NO_ADAPTIVE_MUTEXES. OK, did this recompiling without WITNESS and INVARIANTS and took out ADAPTIVE_GIANT as instructed in the man pages # sysctl -a | grep mutex debug.mutex.prof.enable: 1 debug.mutex.prof.acquisitions: 584672946 debug.mutex.prof.records: 450 debug.mutex.prof.maxrecords: 1000 debug.mutex.prof.rejected: 0 debug.mutex.prof.hashsize: 1009 debug.mutex.prof.collisions: 75 debug.mutex.prof.stats: 10 1263 226 500 /usr/src/sys/kern/sys_pipe.c:1345 (pipe mutex) 3 682 369 1 13 6908 /usr/src/sys/vm/vm_fault.c:851 (vm page queue mutex) 10 2256 623 3 23 15 /usr/src/sys/i386/i386/pmap.c:1891 (vm page queue mutex) 120 883 401 2 15 26 /usr/src/sys/vm/vm_fault.c:909 (vm page queue mutex) 25 25 12500 /usr/src/sys/i386/i386/pmap.c:2572 (vm page queue mutex) 19 169 33 511 /usr/src/sys/vm/vm_object.c:1801 (vm page queue mutex) 4 24 9 200 /usr/src/sys/vm/vm_object.c:646 (vm page queue mutex) 3 23 8 211 /usr/src/sys/kern/vfs_bio.c:3384 (vm page queue mutex) 2 12 6 200 /usr/src/sys/vm/vm_pageout.c:1511 (vm page queue mutex) 4 120 44 200 /usr/src/sys/kern/vfs_bio.c:3315 (vm page queue mutex) 3 100 44 200 /usr/src/sys/kern/vfs_bio.c:3120 (vm page queue mutex) 39 4 200 /usr/src/sys/kern/vfs_bio.c:3577 (vm page queue mutex) 11 1 104 /usr/src/sys/kern/sys_pipe.c:1270 (pipe mutex) 2 16 8 200 /usr/src/sys/vm/vm_page.c:1481 (vm page queue mutex) 28 4 20
Re: Proposed 6.2 em RELEASE patch
Mike Tancsa wrote: At 01:42 AM 11/11/2006, Scott Long wrote: driver. What will help me is if you can hook up a serial console to your machine and see if it can be made to drop to the debugger while it is under load and otherwise unresponsive. If you can, getting a process dump might help confirm where each CPU is spending its time. ./netblast 192.168.88.218 500 110 1000 I compiled in the various debugging options and on the serial console I get a few Expensive timeout(9) function: 0xc0601e48(0) 0.024135749 s and the serial console One CPU seems to be stuck in softclock. My guess here is that there is significant lock contention. Please try the following: 1. Use addr2line or gdb on the kernel to find out what function is at 0xc0601e48 (the address that was printed above). This address will change every time you recompile the kernel, so you'll need to reacquire it from the console each time. 2. Try compiling in WITNESS and running the test as before, then break into the debugger as before. Run 'show locks'. I'm not sure how fruitful this will be, WITNESS might make it unbearably slow. 3. Turn on MUTEX_PROFILING according to the text in /sys/conf/NOTES and the man page by the same name. 4. Remove ADAPTIVE_GIANT from your config and retest. If that doesn't show a difference, then try setting NO_ADAPTIVE_MUTEXES. Thanks, Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
At 08:45 AM 11/12/2006, Adrian Chadd wrote: Should fiddling with the interrupt-coalescing stuff in the em driver via sysctl be tried? None of the recent tests in reply to your email indicate any particular tx/rx threshold settings. I was using whatever is the default. What would you like me to test ? # sysctl -A dev.em dev.em.0.%desc: Intel(R) PRO/1000 Network Connection Version - 6.2.9 dev.em.0.%driver: em dev.em.0.%location: slot=0 function=0 dev.em.0.%pnpinfo: vendor=0x8086 device=0x105e subvendor=0x8086 subdevice=0x115e class=0x02 dev.em.0.%parent: pci6 dev.em.0.debug_info: -1 dev.em.0.stats: -1 dev.em.0.rx_int_delay: 0 dev.em.0.tx_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_abs_int_delay: 66 dev.em.0.rx_processing_limit: 100 dev.em.1.%desc: Intel(R) PRO/1000 Network Connection Version - 6.2.9 dev.em.1.%driver: em dev.em.1.%location: slot=0 function=1 dev.em.1.%pnpinfo: vendor=0x8086 device=0x105e subvendor=0x8086 subdevice=0x115e class=0x02 dev.em.1.%parent: pci6 dev.em.1.debug_info: -1 dev.em.1.stats: -1 dev.em.1.rx_int_delay: 0 dev.em.1.tx_int_delay: 66 dev.em.1.rx_abs_int_delay: 66 dev.em.1.tx_abs_int_delay: 66 dev.em.1.rx_processing_limit: 100 adrian -- Adrian Chadd - [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
Should fiddling with the interrupt-coalescing stuff in the em driver via sysctl be tried? None of the recent tests in reply to your email indicate any particular tx/rx threshold settings. adrian -- Adrian Chadd - [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
At 01:42 AM 11/11/2006, Scott Long wrote: driver. What will help me is if you can hook up a serial console to your machine and see if it can be made to drop to the debugger while it is under load and otherwise unresponsive. If you can, getting a process dump might help confirm where each CPU is spending its time. ./netblast 192.168.88.218 500 110 1000 I compiled in the various debugging options and on the serial console I get a few Expensive timeout(9) function: 0xc0601e48(0) 0.024135749 s and the serial console telnet> send break Expensive timeout(9) function: 0xc0561444(0xc63f1d80) 0.072485748 s KDB: enter: Line break on console [thread pid 27 tid 100017 ] Stopped at kdb_enter+0x2b: nop db> db> ps pid ppid pgrp uid state wmesg wchancmd 1206 1123 1206 0 R+ ifstat 1155 1154 1155 0 R+ csh 1154 1 1154 0 Ss+ wait 0xc6722218 login 1123 1122 1123 0 S+ pause0xc6722894 csh 1122 1117 1122 1002 S+ wait 0xc6739430 su 1117 1116 1117 1002 Ss+ pause0xc6aa024c csh 1116 1114 1114 1002 R sshd 1114 1028 1114 0 Ss sbwait 0xc6893370 sshd 1112 1 1112 0 Ss+ ttyin0xc65ba810 getty 1 0 Ss+ ttyin0xc65bac10 getty 1110 1 1110 0 Ss+ ttyin0xc65bb010 getty 1109 1 1109 0 Ss+ ttyin0xc65bc410 getty 1108 1 1108 0 Ss+ ttyin0xc65b4010 getty 1107 1 1107 0 Ss+ ttyin0xc65bd010 getty 1106 1 1106 0 Ss+ ttyin0xc65bcc10 getty 1105 1 1105 0 Ss+ ttyin0xc65b2010 getty 1044 1 1044 0 Ss nanslp 0xc076ecac cron 1038 1 103825 Ss pause0xc6ab2aac sendmail 1034 1 1034 0 Rs sendmail 1028 1 1028 0 Ss select 0xc07bc004 sshd 898 1 898 0 Ss bo_wwait 0xc6ac9130 syslogd 846 1 846 0 Ss select 0xc07bc004 devd 445 1 44565 Ss select 0xc07bc004 dhclient 425 143 0 S+ select 0xc07bc004 dhclient 124 1 124 0 Ss pause0xc6739034 adjkerntz 42 0 0 0 SL -0xe8ff9d04 [schedcpu] 41 0 0 0 SL sdflush 0xc07c50f4 [softdepflush] 40 0 0 0 RL [syncer] 39 0 0 0 RL [vnlru] 38 0 0 0 SL psleep 0xc07bc56c [bufdaemon] 37 0 0 0 SL pgzero 0xc07c6064 [pagezero] 36 0 0 0 SL psleep 0xc07c5bb4 [vmdaemon] 35 0 0 0 SL psleep 0xc07c5b70 [pagedaemon] 34 0 0 0 WL [irq1: atkbd0] 33 0 0 0 WL [irq7: ppc0] 32 0 0 0 WL [swi0: sio] 31 0 0 0 RL [acpi_cooling0] 30 0 0 0 SL tzpoll 0xc08cd838 [acpi_thermal] 29 0 0 0 WL [irq19: bge1] 28 0 0 0 WL [irq16: bge0+] 27 0 0 0 RL CPU 1 [irq18: em1] 26 0 0 0 WL [irq17: em0] 25 0 0 0 WL [irq23: nve0] 24 0 0 0 WL [irq22: atapci2] 23 0 0 0 WL [irq21: atapci1] 22 0 0 0 WL [irq15: ata1] 21 0 0 0 WL [irq14: ata0] 20 0 0 0 WL [irq9: acpi0] 9 0 0 0 SL -0xc645f080 [kqueue taskq] 19 0 0 0 WL [swi2: cambio] 8 0 0 0 SL -0xc645f280 [acpi_task_2] 7 0 0 0 SL -0xc645f280 [acpi_task_1] 6 0 0 0 SL -0xc645f280 [acpi_task_0] 18 0 0 0 WL [swi5: +] 5 0 0 0 SL -0xc645f400 [thread taskq] 17 0 0 0 WL [swi6: Giant taskq] 16 0 0 0 WL [swi6: task queue] 15 0 0 0 RL [yarrow] 4 0 0 0 RL [g_down] 3 0 0 0 RL [g_up] 2 0 0 0 RL [g_event] 14 0 0 0 WL [swi3: vm] 13 0 0 0 RL CPU 0 [swi4: clock sio] 12 0 0 0 WL [swi1: net] 11 0 0 0 RL [idle: cpu0] 10 0 0 0 RL [idle: cpu1] 1 0 1
Re: Proposed 6.2 em RELEASE patch
At 01:42 AM 11/11/2006, Scott Long wrote: surprised by your results. I'm still a bit unclear on the exact topology of your setup, so if could explain it some more in private email, I'd appreciate it. Hi, I made a quick diagram of the test setup that should make it more clear http://www.tancsa.com/blast.jpg Basically 5 boxes (plus my workstation for out of band access), the main one being tested is the box marked R2 which has a 2 port PCIe em NIC (Pro 1000PT) in the motherboard's 4X slot. I have 2 test boxes as UDP senders and 2 test boxes as UDP receivers, and all the packets flow through the 2 interfaces of R2. With one stream of packets being blasted across, the box is dropping some packets even on its OOB management interface. With 2, its totally unresponsive. Only with polling am I able to continue to work on the box via the OOB interface while one and even 2 streams of UDP packets are blasting across. However, in polling mode some amount of packets are being dropped and I guess I need to better understand how many. My goal in all this is to have a firewall / router that can withstand a high pps workload that will still be reachable OOB when under attack or even under high workload. To measure how many packets are dropped I was looking at making a modified netreceive to count the packets it gets so I can test to see if polling mode will be adequate for my needs. Lets say the max pps the box can handle is X, either in polling or non polling modes. As the box approaches X and gets pushed beyond X, I guess the ideal situation for my needs would be that it drops some packets on the busiest interface so that it can still function and service its other needs, be that network, disk, whatever. But my question is, is X the same for polling and non polling modes. For the short term, I don't think that there is anything that can be magically tweaked that will safely give better results. I know that Gleb has some ideas on a fairly simple change for the non-INTR_FAST, non-POLLING case, but I and several others worry that it's not robust in the face of real-world network problems. For the long term, I have a number of ideas for improving both the RX and TX paths in the driver. Some of it is specific to the if_em driver, some involve improvements in the FFWD and PFIL_HOOKS code as well as the driver. What will help me is if you can hook up a serial console to your machine and see if it can be made to drop to the debugger while it is under load and otherwise unresponsive. If you can, getting a process dump might help confirm where each CPU is spending its time. Yes, I will see what I can do over the weekend. I have some changes to babysit again tomorrow night and will see what I can do between cycles. ---Mike ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
- Original Message - From: "Scott Long" <[EMAIL PROTECTED]> To: "Mike Tancsa" <[EMAIL PROTECTED]> Cc: "freebsd-net" ; <[EMAIL PROTECTED]>; ; "Jack Vogel" <[EMAIL PROTECTED]> Sent: Saturday, November 11, 2006 8:42 AM Subject: Re: Proposed 6.2 em RELEASE patch Mike Tancsa wrote: At 05:00 PM 11/10/2006, Jack Vogel wrote: On 11/10/06, Mike Tancsa <[EMAIL PROTECTED]> wrote: Some more tests. I tried again with what was committed to today's RELENG_6. I am guessing its pretty well the same patch. Polling is the only way to avoid livelock at a high pps rate. Does anyone know of any simple tools to measure end to end packet loss ? Polling will end up dropping some packets and I want to be able to compare. Same hardware from the previous post. The commit WAS the last patch I posted. SO, making sure I understood you, you are saying that POLLING is doing better than FAST_INTR, or only better than the legacy code that went in with my merge? Hi, The last set of tests I posted are ONLY with what is in today's RELENG_6-- i.e. the latest commit. I did a few variations on the driver-- first with #define EM_FAST_INTR 1 in if_em.c one without and one with polling in the kernel. With a decent packet rate passing through, the box will lockup. Not sure if I am just hitting the limits of the PCIe bus, or interrupt moderation is not kicking in, or this is a case of "Doctor, it hurts when I send a lot of packets through"... "Well, dont do that" Using polling prevents the lockup, but it will of course drop packets. This is for firewalls with a fairly high bandwidth rate, as well as I need it to be able to survive a decent DDoS attack. I am not looking for 1Mpps, but something more than 100Kpps ---Mike Hi, Thanks for all of the data. I know that a good amount of testing was done with single stream stress tests, but it's not clear how much was done with multiple streams prior to your efforts. So, I'm not terribly surprised by your results. I'm still a bit unclear on the exact topology of your setup, so if could explain it some more in private email, I'd appreciate it. For the short term, I don't think that there is anything that can be magically tweaked that will safely give better results. I know that Gleb has some ideas on a fairly simple change for the non-INTR_FAST, non-POLLING case, but I and several others worry that it's not robust in the face of real-world network problems. For the long term, I have a number of ideas for improving both the RX and TX paths in the driver. Some of it is specific to the if_em driver, some involve improvements in the FFWD and PFIL_HOOKS code as well as the driver. What will help me is if you can hook up a serial console to your machine and see if it can be made to drop to the debugger while it is under load and otherwise unresponsive. If you can, getting a process dump might help confirm where each CPU is spending its time. Scott I applied Jack's patch to the em driver and all seemed well until xl was giving me the same issues. Thanks Jack on my machine your first patch looks 100% Since my box does not take too much load and to me a slightly more loaded machine is better than an unstable one i re-complied the kernel without SMP so I have a dual CPU system with only one of the CPU's working. I've smacked it with about 50G of data using samba and FTP and it didn't blink. I am however using a fxp card for the live IP side but the xl's are still in the kernel and getting picked up. I have just not configured them with IP's for traffic. I don't think this is the issue tho. I'd say there's something to do with the SMP code that is causing these issues. I have another box with SMP on it. Same kind of setup with a Tyan Tiger instead of a Thunder motherboard. 2 Fxp NICs in it. Most of the time it's stable but if i throw a lot of traffic at it it locks up too. Next time it does I will post the console message, but there is no warnings about watchdog timeouts far as I can remember. It's running 5.5-RELEASE-p8 with SMP enabled. -Clay ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
Mike Tancsa wrote: At 05:00 PM 11/10/2006, Jack Vogel wrote: On 11/10/06, Mike Tancsa <[EMAIL PROTECTED]> wrote: Some more tests. I tried again with what was committed to today's RELENG_6. I am guessing its pretty well the same patch. Polling is the only way to avoid livelock at a high pps rate. Does anyone know of any simple tools to measure end to end packet loss ? Polling will end up dropping some packets and I want to be able to compare. Same hardware from the previous post. The commit WAS the last patch I posted. SO, making sure I understood you, you are saying that POLLING is doing better than FAST_INTR, or only better than the legacy code that went in with my merge? Hi, The last set of tests I posted are ONLY with what is in today's RELENG_6-- i.e. the latest commit. I did a few variations on the driver-- first with #define EM_FAST_INTR 1 in if_em.c one without and one with polling in the kernel. With a decent packet rate passing through, the box will lockup. Not sure if I am just hitting the limits of the PCIe bus, or interrupt moderation is not kicking in, or this is a case of "Doctor, it hurts when I send a lot of packets through"... "Well, dont do that" Using polling prevents the lockup, but it will of course drop packets. This is for firewalls with a fairly high bandwidth rate, as well as I need it to be able to survive a decent DDoS attack. I am not looking for 1Mpps, but something more than 100Kpps ---Mike Hi, Thanks for all of the data. I know that a good amount of testing was done with single stream stress tests, but it's not clear how much was done with multiple streams prior to your efforts. So, I'm not terribly surprised by your results. I'm still a bit unclear on the exact topology of your setup, so if could explain it some more in private email, I'd appreciate it. For the short term, I don't think that there is anything that can be magically tweaked that will safely give better results. I know that Gleb has some ideas on a fairly simple change for the non-INTR_FAST, non-POLLING case, but I and several others worry that it's not robust in the face of real-world network problems. For the long term, I have a number of ideas for improving both the RX and TX paths in the driver. Some of it is specific to the if_em driver, some involve improvements in the FFWD and PFIL_HOOKS code as well as the driver. What will help me is if you can hook up a serial console to your machine and see if it can be made to drop to the debugger while it is under load and otherwise unresponsive. If you can, getting a process dump might help confirm where each CPU is spending its time. Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
At 05:00 PM 11/10/2006, Jack Vogel wrote: On 11/10/06, Mike Tancsa <[EMAIL PROTECTED]> wrote: Some more tests. I tried again with what was committed to today's RELENG_6. I am guessing its pretty well the same patch. Polling is the only way to avoid livelock at a high pps rate. Does anyone know of any simple tools to measure end to end packet loss ? Polling will end up dropping some packets and I want to be able to compare. Same hardware from the previous post. The commit WAS the last patch I posted. SO, making sure I understood you, you are saying that POLLING is doing better than FAST_INTR, or only better than the legacy code that went in with my merge? Hi, The last set of tests I posted are ONLY with what is in today's RELENG_6-- i.e. the latest commit. I did a few variations on the driver-- first with #define EM_FAST_INTR 1 in if_em.c one without and one with polling in the kernel. With a decent packet rate passing through, the box will lockup. Not sure if I am just hitting the limits of the PCIe bus, or interrupt moderation is not kicking in, or this is a case of "Doctor, it hurts when I send a lot of packets through"... "Well, dont do that" Using polling prevents the lockup, but it will of course drop packets. This is for firewalls with a fairly high bandwidth rate, as well as I need it to be able to survive a decent DDoS attack. I am not looking for 1Mpps, but something more than 100Kpps ---Mike ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
On 11/10/06, Mike Tancsa <[EMAIL PROTECTED]> wrote: Some more tests. I tried again with what was committed to today's RELENG_6. I am guessing its pretty well the same patch. Polling is the only way to avoid livelock at a high pps rate. Does anyone know of any simple tools to measure end to end packet loss ? Polling will end up dropping some packets and I want to be able to compare. Same hardware from the previous post. The commit WAS the last patch I posted. SO, making sure I understood you, you are saying that POLLING is doing better than FAST_INTR, or only better than the legacy code that went in with my merge? Jack ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
At 05:00 PM 11/9/2006, Mike Tancsa wrote: At 10:51 AM 11/9/2006, Scott Long wrote: Mike Tancsa wrote: At 08:19 PM 11/8/2006, Jack Vogel wrote: BUT, I've added the FAST_INTR changes back into the code, so if you go into your Makefile and add -DEM_FAST_INTR you will then get the taskqueue stuff. It certainly does make a difference performance wise. I did some quick testing with netperf and netrate. Back to back boxes, using an AMD x2 with bge nic and one intel box CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ (2009.27-MHz 686-class CPU) CPU: Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz (2144.01-MHz 686-class CPU) The intel is a DG965SS with integrated em nic, the AMD a Tyan with integrated bge. Both running SMP kernels with pf built in, no inet6. Intel box as sender. In this test its with the patch from yesterday. The first set with the patch as is, the second test with -DEM_FAST_INTR. Thanks for the tests. One thing to note is that Gleb reported a higher rate of dropped packets with INTR_FAST. He is the only one who has reported this, so I'd like to find out if there is something unique to his environment, or if there is a larger problem to be addressed. There are ways that we can change the driver to not drop any packets at all for Gleb, but they expose the system to risk if there is ever an accidental (or malicious) RX flood on the interface. With a high rate of packets, I am able to live lock the box. I setup the following Some more tests. I tried again with what was committed to today's RELENG_6. I am guessing its pretty well the same patch. Polling is the only way to avoid livelock at a high pps rate. Does anyone know of any simple tools to measure end to end packet loss ? Polling will end up dropping some packets and I want to be able to compare. Same hardware from the previous post. SMP kernel fastfwd pf ipfw FAST_INTR streams np (Mb) x x x 2 livelock x x xx 2 468 livelock x x 2 453 lost packets, box sluggish x xx 2 lost packets, box sluggish x 2 468 lost packets, box sluggish x x 2 468 livelock x x x 2 468 livelock 2 475 livelock x 2 livelock P 2 OK P x 2 OK P x 2 OK The P is for Uniproc, but with Polling enabled (also kern.polling.idle_poll=1) UP single stream 58Kpps, no polling in kernel [bsd6-32bit]# ./netblast 192.168.44.1 500 10 10 start: 1163184051.627479975 finish:1163184061.628200458 send calls:5869051 send errors: 0 approx send rate: 586905 approx error rate: 0 with polling [bsd6-32bit]# ./netblast 192.168.44.1 500 10 10 start: 1163184606.651001121 finish:1163184616.651288588 send calls:5866199 send errors: 1 approx send rate: 586619 approx error rate: 0 With polling and 2 streams at the same time (a lot of pps! and its still totally responsive!!) [r6-32bit]# ./netblast 192.168.88.218 500 10 10 start: 1163184712.103954688 finish:1163184722.104388542 send calls:4528941 send errors: 0 approx send rate: 452894 approx error rate: 0 [r6-32bit]# [bsd6-32bit]# ./netblast 192.168.44.1 500 10 20 start: 1163184793.172036336 finish:1163184813.173028921 send calls:11550594 send errors: 0 approx send rate: 577529 approx error rate: 0 [bsd6-32bit]# polling, 2 streams at the same time [bsd6-32bit]# ./netblast 192.168.44.1 500 10 20 start: 1163185058.477137404 finish:1163185078.478025226 send calls:11679831 send errors: 0 approx send rate: 583991 approx error rate: 0 [bsd6-32bit]# ./netblast 192.168.44.1 500 10 20 start: 1163185167.969551943 finish:1163185187.970435295 send calls:11706825 send errors: 0 approx send rate: 585341 approx error rate: 0 [bsd6-32bit]# ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
Yes, they are incompatible, I suppose there should be something that makes it impossible to do, but not building should be a clue :) Jack On 11/9/06, Mike Tancsa <[EMAIL PROTECTED]> wrote: At 08:19 PM 11/8/2006, Jack Vogel wrote: >BUT, I've added the FAST_INTR changes back into the code, so >if you go into your Makefile and add -DEM_FAST_INTR you will >then get the taskqueue stuff. Not sure why you would want FAST_INTR and polling in at the same time, but I found that the two are mutually exclusive cd /usr/obj/usr/src/sys/pioneer; MAKEOBJDIRPREFIX=/usr/obj MACHINE_ARCH=i386 MACHINE=i386 CPUTYPE= GROFF_BIN_PATH=/usr/obj/usr/src/tmp/legacy/usr/bin GROFF_FONT_PATH=/usr/obj/usr/src/tmp/legacy/usr/share/groff_font GROFF_TMAC_PATH=/usr/obj/usr/src/tmp/legacy/usr/share/tmac _SHLIBDIRPREFIX=/usr/obj/usr/src/tmp INSTALL="sh /usr/src/tools/install.sh" PATH=/usr/obj/usr/src/tmp/legacy/usr/sbin:/usr/obj/usr/src/tmp/legacy/usr/bin:/usr/obj/usr/src/tmp/legacy/usr/games:/usr/obj/usr/src/tmp/usr/sbin:/usr/obj/usr/src/tmp/usr/bin:/usr/obj/usr/src/tmp/usr/games:/sbin:/bin:/usr/sbin:/usr/bin make KERNEL=kernel all -DNO_MODULES_OBJ cc -c -O -pipe -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -fformat-extensions -std=c99 -g -nostdinc -I- -I. -I/usr/src/sys -I/usr/src/sys/contrib/altq -I/usr/src/sys/contrib/ipfilter -I/usr/src/sys/contrib/pf -I/usr/src/sys/dev/ath -I/usr/src/sys/contrib/ngatm -I/usr/src/sys/dev/twa -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -mno-align-long-strings -mpreferred-stack-boundary=2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -ffreestanding -Werror /usr/src/sys/dev/em/if_em.c /usr/src/sys/dev/em/if_em.c: In function `em_ioctl': /usr/src/sys/dev/em/if_em.c:931: error: `em_poll' undeclared (first use in this function) /usr/src/sys/dev/em/if_em.c:931: error: (Each undeclared identifier is reported only once /usr/src/sys/dev/em/if_em.c:931: error: for each function it appears in.) /usr/src/sys/dev/em/if_em.c: At top level: /usr/src/sys/dev/em/if_em.c:1164: warning: 'em_poll' defined but not used *** Error code 1 Stop in /usr/obj/usr/src/sys/pioneer. *** Error code 1 Stop in /usr/src. *** Error code 1 Stop in /usr/src. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
At 10:51 AM 11/9/2006, Scott Long wrote: Mike Tancsa wrote: At 08:19 PM 11/8/2006, Jack Vogel wrote: BUT, I've added the FAST_INTR changes back into the code, so if you go into your Makefile and add -DEM_FAST_INTR you will then get the taskqueue stuff. It certainly does make a difference performance wise. I did some quick testing with netperf and netrate. Back to back boxes, using an AMD x2 with bge nic and one intel box CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ (2009.27-MHz 686-class CPU) CPU: Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz (2144.01-MHz 686-class CPU) The intel is a DG965SS with integrated em nic, the AMD a Tyan with integrated bge. Both running SMP kernels with pf built in, no inet6. Intel box as sender. In this test its with the patch from yesterday. The first set with the patch as is, the second test with -DEM_FAST_INTR. Thanks for the tests. One thing to note is that Gleb reported a higher rate of dropped packets with INTR_FAST. He is the only one who has reported this, so I'd like to find out if there is something unique to his environment, or if there is a larger problem to be addressed. There are ways that we can change the driver to not drop any packets at all for Gleb, but they expose the system to risk if there is ever an accidental (or malicious) RX flood on the interface. With a high rate of packets, I am able to live lock the box. I setup the following b1a --| b2a -R1 b1b |-b2b R1 has PCIe dual port em PRO/1000 PT em0: port 0x9000-0x901f mem 0xd702-0xd703,0xd700-0xd701 irq 1 7 at device 0.0 on pci6 em0: Ethernet address: 00:15:17:0b:70:98 em0: [FAST] em1: port 0x9400-0x941f mem 0xd704-0xd705,0xd706-0xd707 irq 1 8 at device 0.1 on pci6 em1: Ethernet address: 00:15:17:0b:70:99 em1: [FAST] b1a = 192.168.44.1 onboard bge0 b1b = 192.168.88.218 - onboard em 82547EI Gigabit Ethernet Controller b2a = 192.168.88.176 single port PCIe em0 b2b = 192.168.44.244 onboard em0 (DG965SS) R1 has 192.168.44.223 and 192.168.88.223.Routing across R1 with b1a blasting b1b and b2a blastin b2b with netrate will lock up R1 even though the total throughput is only 500Mb. While on b1a # ./netblast 192.168.88.218 500 10 1000 I see the following on R1 (bge1 is my management interface) R1 # ifstat -b em0 em1 bge1 Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out 273770.1 0.00 0.00 237269.1 1.40 3.51 273509.8 0.00 0.00 237040.2 1.73 2.76 273694.9 0.00 0.00 237202.6 0.94 2.34 274258.6 0.00 0.00 237690.4 1.40 2.34 273623.8 0.00 0.00 237140.7 0.94 2.34 If I start up the netblast on b2b or on b2a (either direction, doesnt matter) R1 locks up. This was with R1 in an SMP config. Without INTR_FAST, it doesnt work as fast, but R1 does not lock up, or at least lock me out of my management interface. ---Mike ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
At 08:19 PM 11/8/2006, Jack Vogel wrote: BUT, I've added the FAST_INTR changes back into the code, so if you go into your Makefile and add -DEM_FAST_INTR you will then get the taskqueue stuff. Not sure why you would want FAST_INTR and polling in at the same time, but I found that the two are mutually exclusive cd /usr/obj/usr/src/sys/pioneer; MAKEOBJDIRPREFIX=/usr/obj MACHINE_ARCH=i386 MACHINE=i386 CPUTYPE= GROFF_BIN_PATH=/usr/obj/usr/src/tmp/legacy/usr/bin GROFF_FONT_PATH=/usr/obj/usr/src/tmp/legacy/usr/share/groff_font GROFF_TMAC_PATH=/usr/obj/usr/src/tmp/legacy/usr/share/tmac _SHLIBDIRPREFIX=/usr/obj/usr/src/tmp INSTALL="sh /usr/src/tools/install.sh" PATH=/usr/obj/usr/src/tmp/legacy/usr/sbin:/usr/obj/usr/src/tmp/legacy/usr/bin:/usr/obj/usr/src/tmp/legacy/usr/games:/usr/obj/usr/src/tmp/usr/sbin:/usr/obj/usr/src/tmp/usr/bin:/usr/obj/usr/src/tmp/usr/games:/sbin:/bin:/usr/sbin:/usr/bin make KERNEL=kernel all -DNO_MODULES_OBJ cc -c -O -pipe -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -fformat-extensions -std=c99 -g -nostdinc -I- -I. -I/usr/src/sys -I/usr/src/sys/contrib/altq -I/usr/src/sys/contrib/ipfilter -I/usr/src/sys/contrib/pf -I/usr/src/sys/dev/ath -I/usr/src/sys/contrib/ngatm -I/usr/src/sys/dev/twa -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -mno-align-long-strings -mpreferred-stack-boundary=2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -ffreestanding -Werror /usr/src/sys/dev/em/if_em.c /usr/src/sys/dev/em/if_em.c: In function `em_ioctl': /usr/src/sys/dev/em/if_em.c:931: error: `em_poll' undeclared (first use in this function) /usr/src/sys/dev/em/if_em.c:931: error: (Each undeclared identifier is reported only once /usr/src/sys/dev/em/if_em.c:931: error: for each function it appears in.) /usr/src/sys/dev/em/if_em.c: At top level: /usr/src/sys/dev/em/if_em.c:1164: warning: 'em_poll' defined but not used *** Error code 1 Stop in /usr/obj/usr/src/sys/pioneer. *** Error code 1 Stop in /usr/src. *** Error code 1 Stop in /usr/src. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
Mike Tancsa wrote: At 08:19 PM 11/8/2006, Jack Vogel wrote: BUT, I've added the FAST_INTR changes back into the code, so if you go into your Makefile and add -DEM_FAST_INTR you will then get the taskqueue stuff. It certainly does make a difference performance wise. I did some quick testing with netperf and netrate. Back to back boxes, using an AMD x2 with bge nic and one intel box CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ (2009.27-MHz 686-class CPU) CPU: Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz (2144.01-MHz 686-class CPU) The intel is a DG965SS with integrated em nic, the AMD a Tyan with integrated bge. Both running SMP kernels with pf built in, no inet6. Intel box as sender. In this test its with the patch from yesterday. The first set with the patch as is, the second test with -DEM_FAST_INTR. Thanks for the tests. One thing to note is that Gleb reported a higher rate of dropped packets with INTR_FAST. He is the only one who has reported this, so I'd like to find out if there is something unique to his environment, or if there is a larger problem to be addressed. There are ways that we can change the driver to not drop any packets at all for Gleb, but they expose the system to risk if there is ever an accidental (or malicious) RX flood on the interface. Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
At 08:19 PM 11/8/2006, Jack Vogel wrote: BUT, I've added the FAST_INTR changes back into the code, so if you go into your Makefile and add -DEM_FAST_INTR you will then get the taskqueue stuff. It certainly does make a difference performance wise. I did some quick testing with netperf and netrate. Back to back boxes, using an AMD x2 with bge nic and one intel box CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ (2009.27-MHz 686-class CPU) CPU: Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz (2144.01-MHz 686-class CPU) The intel is a DG965SS with integrated em nic, the AMD a Tyan with integrated bge. Both running SMP kernels with pf built in, no inet6. Intel box as sender. In this test its with the patch from yesterday. The first set with the patch as is, the second test with -DEM_FAST_INTR. TCP STREAM TEST to 192.168.44.1 : +/-2.5% @ 99% conf. Recv SendSend Socket Socket Message Elapsed Size SizeSize Time Throughput bytes bytes bytessecs.10^6bits/sec 57344 57344 409662.19 858.16 57344 57344 409662.19 934.58 TCP STREAM TEST to 192.168.44.1 : +/-2.5% @ 99% conf. Recv SendSend Socket Socket Message Elapsed Size SizeSize Time Throughput bytes bytes bytessecs.10^6bits/sec 32768 32768 409662.27 551.46 32768 32768 409662.26 788.56 TCP REQUEST/RESPONSE TEST to 192.168.44.1 : +/-2.5% @ 99% conf. Local /Remote Socket Size Request Resp. Elapsed Trans. Send Recv Size SizeTime Rate bytes Bytes bytesbytes secs.per sec 32768 65536 11 62.262999.88 32768 65536 32768 65536 11 62.316165.46 32768 65536 UDP REQUEST/RESPONSE TEST to 192.168.44.1 : +/-2.5% @ 99% conf. Local /Remote Socket Size Request Resp. Elapsed Trans. Send Recv Size SizeTime Rate bytes Bytes bytesbytes secs.per sec 9216 41600 11 62.303170.25 9216 41600 9216 41600 11 62.346170.81 9216 41600 UDP REQUEST/RESPONSE TEST to 192.168.44.1 : +/-2.5% @ 99% conf. Local /Remote Socket Size Request Resp. Elapsed Trans. Send Recv Size SizeTime Rate bytes Bytes bytesbytes secs.per sec 9216 41600 516 4 62.282999.17 9216 41600 9216 41600 516 4 62.336031.56 9216 41600 UDP UNIDIRECTIONAL SEND TEST to 192.168.44.1 : +/-2.5% @ 99% conf. Socket Message Elapsed Messages SizeSize Time Okay Errors Throughput bytes bytessecs# # 10^6bits/sec 327684096 60.00 1743632 24778919 952.25 41600 60.00 1742801951.79 327684096 60.00 1743633 24722456 952.25 41600 60.00 1742828951.81 UDP UNIDIRECTIONAL SEND TEST to 192.168.44.1 : +/-2.5% @ 99% conf. Socket Message Elapsed Messages SizeSize Time Okay Errors Throughput bytes bytessecs# # 10^6bits/sec 327681024 60.00 6831370 28639884 932.70 41600 60.00 6828166932.27 327681024 60.00 6831369 28465662 932.70 41600 60.00 6828086932.26 Intel box as receiver, bge0/AMD as sender First set of results using stock em driver from 6.2beta2 second set of results using first patch 3rd set using taskqueue enabled /usr/local/netperf/netperf -t TCP_STREAM -l 60 -H 192.168.44.244 -i 10,3 -I 99,5 -- -s 57344 -S 57344 -m 4096 TCP STREAM TEST to 192.168.44.244 : +/-2.5% @ 99% conf. Recv SendSend Socket Socket Message Elapsed Size SizeSize Time Throughput bytes bytes bytessecs.10^6bits/sec 57344 57344 409660.00 680.24 57344 57344 409660.00 680.34 57344 57344 409660.00 680.54 TCP STREAM TEST to 192.168.44.244 : +/-2.5% @ 99% conf. Recv SendSend Socket Socket Message Elapsed Size SizeSize Time Throughput bytes bytes bytessecs.10^6bits/sec 32768 32768 409660.00 496.72 32768 32768 409660.00 499.87 32768 32768 409660.00 677.63 TCP REQUEST/RESPONSE TEST to 192.168.44.244 : +/-2.5% @ 99% conf. Local /Remote Socket Size Request Resp. Elapsed Trans. Send Recv Size SizeTime Rate bytes Bytes bytesbytes secs.per sec 32768 65536 11 60.002999.61 32768 65536 32768 65536 11 60.002999.50 32768 65536 32768 65536 11 60.006163.75 32768 65536 UDP REQUEST/RESPONSE TEST to 192.168.44.244 : +/-2.5% @ 99% conf. Local /Remote Socket Size Request Resp. Elapsed Trans. Send Recv Size SizeTime Rate bytes Bytes bytesbytes secs.per sec 9216 41600 11 60.003099.52 9216 41600 9216 41600 11 60.003102.97 9216 41600 9216 41600 1
Re: Proposed 6.2 em RELEASE patch
On 11/8/06, Sam Wun <[EMAIL PROTECTED]> wrote: Without introduced this new patch, can I still use sysctl to maximise its performance like FAST_INTR? S Not sure if I'm understanding you, but let me try. You cannot attain the same receive performance without the patch. When I use SmartBits and blast UDP packets at TWO fiber PCI-E NICS and set it to 70% utilization of the line it will just BURY the system, meaning that the shell on the console will appear wedged. Once you stop the test it recovers, but during it its totally consumed handling interrupts. Perhaps with careful tweaking of everything you can make things better, but if so that goes beyond my tuning knowledge. Just one NIC will be OK, and if I drop the utilization down to 45% its ok, but 50 and up and we go into the tank, as it were :) If you compile with EM_FAST_INTR then the system will continue to operate quite well with the same load. Now, this is one kind of load, and there is still other types that work just fine without FAST_INTR, and without the patch you can still use sysctl to adjust tuning parameters as your needs vary. BUT, I do not believe you can do as well as you can with FAST_INTR on, this is why I wanted to get this code back into the driver conditionally before RELEASE. Does that answer your question Sam? Regards, Jack On 11/9/06, Jack Vogel < [EMAIL PROTECTED]> wrote: > > This patch is an evolution of the last one I sent out. It has > a couple of minor corrections, like a bad forward decl in > the header. > > The last patch has had quite a bit of testing and all reports > have been positive. The only complaint was from Gleb who > says he needs to keep his beloved infinite for loop in the > interrupt handler, well I have a better one for you Gleb, keep > reading. > > I have also been doing some extreme stress testing using > SmartBits, and discovered the driver as it stands is really > not able to take extreme receive side pounding, Scott > pointed out that this is why the FAST_INTR work was done :) > > There were some people that had stability issues with that > work, but there were also many that did not. I actually > merged the FAST code onto my last patch, and ran the > SB stress and found it really was able to gracefully handle > that load, way to go Scott :) > > I've pondered this situation, and this patch I'm including here > today is the result. Here's what it does: > > If you drop it in place, compile it, and go... you will get the > code that has been tested for a week, it uses the older > style interrupts, it has the watchdog and other SMP fixes > so its been proven. > > BUT, I've added the FAST_INTR changes back into the code, so > if you go into your Makefile and add -DEM_FAST_INTR you will > then get the taskqueue stuff. > > So, Gleb, rather than replace the infinite for loop that no one > thinks is a good idea, you can just define FAST_INTR again, > and you should be good to go. > > I see this as the best thing for the 6.2 RELEASE, it lets us > keep moving forward, people that want max performance > can define EM_FAST_INTR and help us wring out any > problems, it also will mean that I will have our Intel test > group start using this code. But for those that just want > a stable driver the standard compile will still give them that. > > The patch I'm including is against BETA3. Let me know of > your concerns or issues. > > Cheers, > > Jack > > > ___ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to " [EMAIL PROTECTED]" > > > ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Proposed 6.2 em RELEASE patch
Without introduced this new patch, can I still use sysctl to maximise its performance like FAST_INTR? S On 11/9/06, Jack Vogel <[EMAIL PROTECTED]> wrote: This patch is an evolution of the last one I sent out. It has a couple of minor corrections, like a bad forward decl in the header. The last patch has had quite a bit of testing and all reports have been positive. The only complaint was from Gleb who says he needs to keep his beloved infinite for loop in the interrupt handler, well I have a better one for you Gleb, keep reading. I have also been doing some extreme stress testing using SmartBits, and discovered the driver as it stands is really not able to take extreme receive side pounding, Scott pointed out that this is why the FAST_INTR work was done :) There were some people that had stability issues with that work, but there were also many that did not. I actually merged the FAST code onto my last patch, and ran the SB stress and found it really was able to gracefully handle that load, way to go Scott :) I've pondered this situation, and this patch I'm including here today is the result. Here's what it does: If you drop it in place, compile it, and go... you will get the code that has been tested for a week, it uses the older style interrupts, it has the watchdog and other SMP fixes so its been proven. BUT, I've added the FAST_INTR changes back into the code, so if you go into your Makefile and add -DEM_FAST_INTR you will then get the taskqueue stuff. So, Gleb, rather than replace the infinite for loop that no one thinks is a good idea, you can just define FAST_INTR again, and you should be good to go. I see this as the best thing for the 6.2 RELEASE, it lets us keep moving forward, people that want max performance can define EM_FAST_INTR and help us wring out any problems, it also will mean that I will have our Intel test group start using this code. But for those that just want a stable driver the standard compile will still give them that. The patch I'm including is against BETA3. Let me know of your concerns or issues. Cheers, Jack ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
