Re: High CPU interrupt load on intel I350T4 with igb on 8.3
On Friday, April 26, 2013 7:31:07 am Clément Hermann (nodens) wrote: > Hi list, > > We use pf+ALTQ for trafic shaping on some routers. > > We are switching to new servers : Dell PowerEdge R620 with 2 8-cores > Intel Processor (E5-2650L), 8GB RAM and Intel I350T4 (quad port) using > igb driver. The old hardware is using em driver, the CPU load is high > but mostly due to kernel and a large pf ruleset. > > On the new hardware, we see high CPU Interrupt load (up to 95%), even > though there is not much trafic currently (peaks about 150Mbps and > 40Kpps). All queues are used and binded to a cpu according to top, but a > lot of CPU time is spent on igb queues (interrupt or wait). The load is > fine when we stay below 20Kpps. > > We see no mbuf shortage, no dropped packet, but there is little margin > left on CPU time (about 25% idle at best, most of CPU time is spent on > interrupts), which is disturbing. > > We have done some tuning, but to no avail : If you have the processing_limit set to -1, you should never see CPU time spent in the igb task threads (any such time means there is a bug). One such bug was fixed in 8.x here (that is after 8.3): http://svnweb.freebsd.org/base?view=revision&revision=235553 This may not help with any issues in pf(4), but we had workloads at work (not involving pf) where this bug could cause boxes to spend 100% CPU in igb threads. -- John Baldwin ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: High CPU interrupt load on intel I350T4 with igb on 8.3
On 14.05.2013 08:08, Barney Cordoba wrote: > The defaults are broken. Finally we managed to get to the point. Hallelujah! ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: High CPU interrupt load on intel I350T4 with igb on 8.3
You have to admit there's a problem before you can fix it. If Eugene is going to blame to bottleneck and no one is going to tell him he's wrong, then there is no discussion. The solution in this case is to use 1 queue, which was my suggestion many days ago. The defaults are broken. The driver should default to 1 queue, and be tuned to the system environment. With 2 NICs in the box, the defaults are defective. 1 queue should always work. Other settings require tuning and an understanding of how things work. I've had to support i350 so I've been playing with the driver a bit. It works fine with lots of cores. But you have to have more cores than queues. 2 cards with 4 queues on a 6 physical core system gets into a contention problem at certain loads. I've also removed the cpu bindings, which is about all I'm free to disclose. The driver needs a tuning doc as much as anything else. BC --- On Sat, 5/11/13, Adrian Chadd wrote: > From: Adrian Chadd > Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3 > To: "Hooman Fazaeli" > Cc: "Barney Cordoba" , ""Clément Hermann (nodens)"" > , "Eugene Grosbein" , > freebsd-net@freebsd.org > Date: Saturday, May 11, 2013, 6:16 PM > Hi, > > The motivation behind the locking scheme in igb in friends > is for a > very specific, userland-traffic-origin workload. > > Sure, it may or may not work well for forwarding/filtering > workloads. > > If you want to fix it, let's have a discussion about how to > do it, > followed by some patches to do so. > > > > > Adrian > > On 11 May 2013 13:12, Hooman Fazaeli > wrote: > > On 5/11/2013 8:26 PM, Barney Cordoba wrote: > >> Clearly you don't understand the problem. Your > logic is that because other drivers are defective also; > therefore its not a driver problem? The problem is caused by > a multi-threaded driver that > >> haphazardly launches tasks and that doesn't manage > the case that the rest of the system can't handle the load. > It's no different than a driver that barfs when mbuf > clusters are exhausted. The answer > >> isn't to increase memory or mbufs, even though that > may alleviate the problem. The answer is to fix the driver, > so that it doesn't crash the system for an event that is > wholly predictable. igb has > >> 1) too many locks and 2) exasperates the problem by > binding to cpus, which causes it to not only have to wait > for the lock to free, but also for a specific cpu to become > free. So it chugs along > >> happily until it encounters a bottleneck, at which > point it quickly blows up the entire system in a domino > effect. It needs to manage locks more efficiently, and also > to detect when the backup is > >> unmanageable. Ever since FreeBSD 5 the answer has > been "it's fixed in 7, or its fixed in 9, or it's fixed in > 10". There will always be bottlenecks, and no driver should > blow up the system no matter > >> what intermediate code may present a problem. Its > the driver's responsibility to behave and to drop packets if > necessary. BC > > > > And how the driver should behave? You suggest dropping > the packets. Even if we accept > > that dropping packets is a good strategy in all > configurations (which I doubt), the driver is > > definitely not the best place to implement it, since > that involves duplication of similar > > code between drivers. Somewhere like the Ethernet layer > is a much better choice to watch > > load of packets and drop them to prevent them to eat > all the cores. Furthermore, ignoring > > the fact that pf is not optimized for multi-processors > and blaming drivers for not adjusting > > themselves with the this pf's fault, is a bit unfair, I > believe. > > > > > > -- > > > > Best regards. > > Hooman Fazaeli > > > > ___ > > freebsd-net@freebsd.org > mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" > ___ > freebsd-net@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" > ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: High CPU interrupt load on intel I350T4 with igb on 8.3
On May 11, 2013, at 2:12 PM, Hooman Fazaeli wrote: > On 5/11/2013 8:26 PM, Barney Cordoba wrote: >> Clearly you don't understand the problem. Your logic is that because other >> drivers are defective also; therefore its not a driver problem? The problem >> is caused by a multi-threaded driver that >> haphazardly launches tasks and that doesn't manage the case that the rest of >> the system can't handle the load. It's no different than a driver that barfs >> when mbuf clusters are exhausted. The answer >> isn't to increase memory or mbufs, even though that may alleviate the >> problem. The answer is to fix the driver, so that it doesn't crash the >> system for an event that is wholly predictable. igb has >> 1) too many locks and 2) exasperates the problem by binding to cpus, which >> causes it to not only have to wait for the lock to free, but also for a >> specific cpu to become free. So it chugs along >> happily until it encounters a bottleneck, at which point it quickly blows up >> the entire system in a domino effect. It needs to manage locks more >> efficiently, and also to detect when the backup is >> unmanageable. Ever since FreeBSD 5 the answer has been "it's fixed in 7, or >> its fixed in 9, or it's fixed in 10". There will always be bottlenecks, and >> no driver should blow up the system no matter >> what intermediate code may present a problem. Its the driver's >> responsibility to behave and to drop packets if necessary. BC > > And how the driver should behave? You suggest dropping the packets. Even if > we accept > that dropping packets is a good strategy in all configurations (which I > doubt), the driver is > definitely not the best place to implement it, since that involves > duplication of similar > code between drivers. Somewhere like the Ethernet layer is a much better > choice to watch > load of packets and drop them to prevent them to eat all the cores. > Furthermore, ignoring > the fact that pf is not optimized for multi-processors and blaming drivers > for not adjusting > themselves with the this pf's fault, is a bit unfair, I believe. > Fortunately I no longer receive Barney's emails, but it still distresses me to see him trolling the list. It should be a pretty big hint that Barney has nothing to offer the conversation when he suggests on a technical level that dropping packets is an acceptable policy for drivers. The conversation is also over when he resorts to the ad hominem attacks and the blanket "driver X sucks and you all are too lazy to follow my brilliance in fixing it" tripe. Can we all just put him on ignore and move on? A brief search of the PR database shows no contributions from him. A brief search of the mailing lists shows only inflamed diatribes and insults from him, with a brief smattering here and there of benign but content-free posts. On the other hand, if the consensus here is to keep on baiting and feeding him for our own amusement, I applaud the effort but ask for a bit more subtlety =-) Thanks! Scott ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: High CPU interrupt load on intel I350T4 with igb on 8.3
On 11.05.2013 22:56, Barney Cordoba wrote: >> In practice, the problem is easily solved without any change >> in the igb code. >> The same problem will occur for other NIC drivers too - >> if several NICs were combined within one lagg(4). So, driver >> is not guilty and >> solution would be same - eliminate bottleneck and you will >> be fine and capable >> to spread the load on several CPU cores. >> >> Therefore, I don't care of CS theory for this particular >> case. > > Clearly you don't understand the problem. Your logic is that because > other drivers are defective also; therefore its not a driver problem? The problem is not in drivers. Driver author should make no assumption of manner packets will be processed after NIC because there are pretty many patterns. NIC driver must get packets from the NIC with minimal overhead, loading so many CPU cores as it instructed to by system settings (loader.conf/sysctl.conf/etc.) and never do voluntary packet drops. > The problem is caused by a multi-threaded driver that haphazardly launches > tasks and that doesn't manage the case that the rest of the system can't > handle the load. This is not driver author who decides how many tasks to launch and what will happen with packets later. I, as system administrator, perform driver and system tuning for my particular task and I decide that, using system settings. > It's no different than a driver that barfs when mbuf > clusters are exhausted. The answer isn't to increase memory or mbufs, even > though that may alleviate the problem. The answer is to fix the driver, > so that it doesn't crash the system for an event that is wholly predictable. No crashes observed due to igb(4) and it is easy to make the system predictable with some tuning. > igb has 1) too many locks and 2) exasperates the problem by binding to > cpus, which causes it to not only have to wait for the lock to free, but > also for a specific cpu to become free. I can't say for locks but CPU binding is not a problem. One always can use cpuset(1) to make bindings suitable for particular task. For example, this is rcNG script I use for my igb-based BRAS'es to change default igb bindings: #!/bin/sh # PROVIDE: cpuset-igb # REQUIRE: FILESYSTEMS # BEFORE: netif # KEYWORD: nojail case "$1" in *start) echo "Binding igb(4) IRQs to CPUs" cpus=`sysctl -n kern.smp.cpus` vmstat -ai | sed -E '/^irq.*que/!d; s/^irq([0-9]+): igb([0-9]+):que ([0-9]+).*/\1 \2 \3/' |\ while read irq igb que do cpuset -l $(( ($igb+$que) % $cpus )) -x $irq done ;; esac There is no rocket science here. Even simplier script may be used to disable CPU bindings altogether using "cpuset -l all" command. > So it chugs along happily until > it encounters a bottleneck, at which point it quickly blows up the entire > system in a domino effect. It needs to manage locks more efficiently, and > also to detect when the backup is unmanageable. > Ever since FreeBSD 5 the answer has been "it's fixed in 7, or its fixed in > 9, or it's fixed in 10". There will always be bottlenecks, and no driver > should blow up the system no matter what intermediate code may present a > problem. Its the driver's responsibility to behave and to drop packets > if necessary. Voluntary packet drops at driver level should not be permitted. That's not driver's responsibility. I, as system administrator, expect from a driver to spit packets as fast as possible. I can employ some packet buffering system if I need it or use ng_car or dummynet or something to deal with bottlenecks. Or not to deal with them for purpose. NIC driver cannot know my needs. Eugene Grosbein ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: High CPU interrupt load on intel I350T4 with igb on 8.3
--- On Sat, 5/11/13, Hooman Fazaeli wrote: > From: Hooman Fazaeli > Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3 > To: "Barney Cordoba" > Cc: "Eugene Grosbein" , freebsd-net@freebsd.org, ""Clément > Hermann (nodens)"" > Date: Saturday, May 11, 2013, 4:12 PM > On 5/11/2013 8:26 PM, Barney Cordoba > wrote: > > Clearly you don't understand the problem. Your logic is > that because other drivers are defective also; therefore its > not a driver problem? The problem is caused by a > multi-threaded driver that > > haphazardly launches tasks and that doesn't manage the > case that the rest of the system can't handle the load. It's > no different than a driver that barfs when mbuf clusters are > exhausted. The answer > > isn't to increase memory or mbufs, even though that may > alleviate the problem. The answer is to fix the driver, so > that it doesn't crash the system for an event that is wholly > predictable. igb has > > 1) too many locks and 2) exasperates the problem by > binding to cpus, which causes it to not only have to wait > for the lock to free, but also for a specific cpu to become > free. So it chugs along > > happily until it encounters a bottleneck, at which > point it quickly blows up the entire system in a domino > effect. It needs to manage locks more efficiently, and also > to detect when the backup is > > unmanageable. Ever since FreeBSD 5 the answer has been > "it's fixed in 7, or its fixed in 9, or it's fixed in 10". > There will always be bottlenecks, and no driver should blow > up the system no matter > > what intermediate code may present a problem. Its the > driver's responsibility to behave and to drop packets if > necessary. BC > > And how the driver should behave? You suggest dropping the > packets. Even if we accept > that dropping packets is a good strategy in all > configurations (which I doubt), the driver is > definitely not the best place to implement it, since that > involves duplication of similar > code between drivers. Somewhere like the Ethernet layer is a > much better choice to watch > load of packets and drop them to prevent them to eat all the > cores. Furthermore, ignoring > the fact that pf is not optimized for multi-processors and > blaming drivers for not adjusting > themselves with the this pf's fault, is a bit unfair, I > believe. > It's easier to make excuses than to write a really good driver. I'll grant you that. BC ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: High CPU interrupt load on intel I350T4 with igb on 8.3
Hi, The motivation behind the locking scheme in igb in friends is for a very specific, userland-traffic-origin workload. Sure, it may or may not work well for forwarding/filtering workloads. If you want to fix it, let's have a discussion about how to do it, followed by some patches to do so. Adrian On 11 May 2013 13:12, Hooman Fazaeli wrote: > On 5/11/2013 8:26 PM, Barney Cordoba wrote: >> Clearly you don't understand the problem. Your logic is that because other >> drivers are defective also; therefore its not a driver problem? The problem >> is caused by a multi-threaded driver that >> haphazardly launches tasks and that doesn't manage the case that the rest of >> the system can't handle the load. It's no different than a driver that barfs >> when mbuf clusters are exhausted. The answer >> isn't to increase memory or mbufs, even though that may alleviate the >> problem. The answer is to fix the driver, so that it doesn't crash the >> system for an event that is wholly predictable. igb has >> 1) too many locks and 2) exasperates the problem by binding to cpus, which >> causes it to not only have to wait for the lock to free, but also for a >> specific cpu to become free. So it chugs along >> happily until it encounters a bottleneck, at which point it quickly blows up >> the entire system in a domino effect. It needs to manage locks more >> efficiently, and also to detect when the backup is >> unmanageable. Ever since FreeBSD 5 the answer has been "it's fixed in 7, or >> its fixed in 9, or it's fixed in 10". There will always be bottlenecks, and >> no driver should blow up the system no matter >> what intermediate code may present a problem. Its the driver's >> responsibility to behave and to drop packets if necessary. BC > > And how the driver should behave? You suggest dropping the packets. Even if > we accept > that dropping packets is a good strategy in all configurations (which I > doubt), the driver is > definitely not the best place to implement it, since that involves > duplication of similar > code between drivers. Somewhere like the Ethernet layer is a much better > choice to watch > load of packets and drop them to prevent them to eat all the cores. > Furthermore, ignoring > the fact that pf is not optimized for multi-processors and blaming drivers > for not adjusting > themselves with the this pf's fault, is a bit unfair, I believe. > > > -- > > Best regards. > Hooman Fazaeli > > ___ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: High CPU interrupt load on intel I350T4 with igb on 8.3
On 5/11/2013 8:26 PM, Barney Cordoba wrote: > Clearly you don't understand the problem. Your logic is that because other > drivers are defective also; therefore its not a driver problem? The problem > is caused by a multi-threaded driver that > haphazardly launches tasks and that doesn't manage the case that the rest of > the system can't handle the load. It's no different than a driver that barfs > when mbuf clusters are exhausted. The answer > isn't to increase memory or mbufs, even though that may alleviate the > problem. The answer is to fix the driver, so that it doesn't crash the system > for an event that is wholly predictable. igb has > 1) too many locks and 2) exasperates the problem by binding to cpus, which > causes it to not only have to wait for the lock to free, but also for a > specific cpu to become free. So it chugs along > happily until it encounters a bottleneck, at which point it quickly blows up > the entire system in a domino effect. It needs to manage locks more > efficiently, and also to detect when the backup is > unmanageable. Ever since FreeBSD 5 the answer has been "it's fixed in 7, or > its fixed in 9, or it's fixed in 10". There will always be bottlenecks, and > no driver should blow up the system no matter > what intermediate code may present a problem. Its the driver's responsibility > to behave and to drop packets if necessary. BC And how the driver should behave? You suggest dropping the packets. Even if we accept that dropping packets is a good strategy in all configurations (which I doubt), the driver is definitely not the best place to implement it, since that involves duplication of similar code between drivers. Somewhere like the Ethernet layer is a much better choice to watch load of packets and drop them to prevent them to eat all the cores. Furthermore, ignoring the fact that pf is not optimized for multi-processors and blaming drivers for not adjusting themselves with the this pf's fault, is a bit unfair, I believe. -- Best regards. Hooman Fazaeli ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: High CPU interrupt load on intel I350T4 with igb on 8.3
--- On Fri, 5/10/13, Eugene Grosbein wrote: > From: Eugene Grosbein > Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3 > To: "Barney Cordoba" > Cc: freebsd-net@freebsd.org, ""Clément Hermann (nodens)"" > > Date: Friday, May 10, 2013, 8:56 AM > On 10.05.2013 05:16, Barney Cordoba > wrote: > > >>>> Network device driver is not guilty here, > that's > >> just pf's > >>>> contention > >>>> running in igb's context. > >>> > >>> They're both at play. Single threadedness > aggravates > >> subsystems that > >>> have too many lock points. > >>> > >>> It can also be "solved" with using 1 queue, > because > >> then you don't > >>> have 4 queues going into a single thread. > >> > >> Again, the problem is within pf(4)'s global lock, > not in the > >> igb(4). > >> > > > > Again, you're wrong. It's not the bottleneck's fault; > it's the fault of > > the multi-threaded code for only working properly when > there are no > > bottlenecks. > > In practice, the problem is easily solved without any change > in the igb code. > The same problem will occur for other NIC drivers too - > if several NICs were combined within one lagg(4). So, driver > is not guilty and > solution would be same - eliminate bottleneck and you will > be fine and capable > to spread the load on several CPU cores. > > Therefore, I don't care of CS theory for this particular > case. Clearly you don't understand the problem. Your logic is that because other drivers are defective also; therefore its not a driver problem? The problem is caused by a multi-threaded driver that haphazardly launches tasks and that doesn't manage the case that the rest of the system can't handle the load. It's no different than a driver that barfs when mbuf clusters are exhausted. The answer isn't to increase memory or mbufs, even though that may alleviate the problem. The answer is to fix the driver, so that it doesn't crash the system for an event that is wholly predictable. igb has 1) too many locks and 2) exasperates the problem by binding to cpus, which causes it to not only have to wait for the lock to free, but also for a specific cpu to become free. So it chugs along happily until it encounters a bottleneck, at which point it quickly blows up the entire system in a domino effect. It needs to manage locks more efficiently, and also to detect when the backup is unmanageable. Ever since FreeBSD 5 the answer has been "it's fixed in 7, or its fixed in 9, or it's fixed in 10". There will always be bottlenecks, and no driver should blow up the system no matter what intermediate code may present a problem. Its the driver's responsibility to behave and to drop packets if necessary. BC ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: High CPU interrupt load on intel I350T4 with igb on 8.3
On 10.05.2013 05:16, Barney Cordoba wrote: Network device driver is not guilty here, that's >> just pf's contention running in igb's context. >>> >>> They're both at play. Single threadedness aggravates >> subsystems that >>> have too many lock points. >>> >>> It can also be "solved" with using 1 queue, because >> then you don't >>> have 4 queues going into a single thread. >> >> Again, the problem is within pf(4)'s global lock, not in the >> igb(4). >> > > Again, you're wrong. It's not the bottleneck's fault; it's the fault of > the multi-threaded code for only working properly when there are no > bottlenecks. In practice, the problem is easily solved without any change in the igb code. The same problem will occur for other NIC drivers too - if several NICs were combined within one lagg(4). So, driver is not guilty and solution would be same - eliminate bottleneck and you will be fine and capable to spread the load on several CPU cores. Therefore, I don't care of CS theory for this particular case. Eugene Grosbein ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: High CPU interrupt load on intel I350T4 with igb on 8.3
--- On Thu, 5/9/13, Eugene Grosbein wrote: > From: Eugene Grosbein > Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3 > To: "Barney Cordoba" > Cc: ""Clément Hermann (nodens)"" , > freebsd-net@freebsd.org > Date: Thursday, May 9, 2013, 12:30 PM > On 09.05.2013 23:25, Barney Cordoba > wrote: > > >> Network device driver is not guilty here, that's > just pf's > >> contention > >> running in igb's context. > >> > >> Eugene Grosbein > > > > They're both at play. Single threadedness aggravates > subsystems that > > have too many lock points. > > > > It can also be "solved" with using 1 queue, because > then you don't > > have 4 queues going into a single thread. > > Again, the problem is within pf(4)'s global lock, not in the > igb(4). > Again, you're wrong. It's not the bottleneck's fault; it's the fault of the multi-threaded code for only working properly when there are no bottlenecks. BC ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: High CPU interrupt load on intel I350T4 with igb on 8.3
On 09.05.2013 23:25, Barney Cordoba wrote: >> Network device driver is not guilty here, that's just pf's >> contention >> running in igb's context. >> >> Eugene Grosbein > > They're both at play. Single threadedness aggravates subsystems that > have too many lock points. > > It can also be "solved" with using 1 queue, because then you don't > have 4 queues going into a single thread. Again, the problem is within pf(4)'s global lock, not in the igb(4). ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: High CPU interrupt load on intel I350T4 with igb on 8.3
--- On Thu, 5/9/13, Eugene Grosbein wrote: > From: Eugene Grosbein > Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3 > To: ""Clément Hermann (nodens)"" > Cc: freebsd-net@freebsd.org > Date: Thursday, May 9, 2013, 10:55 AM > On 26.04.2013 18:31, "Clément > Hermann (nodens)" wrote: > > Hi list, > > > > We use pf+ALTQ for trafic shaping on some routers. > > > > We are switching to new servers : Dell PowerEdge R620 > with 2 8-cores > > Intel Processor (E5-2650L), 8GB RAM and Intel I350T4 > (quad port) using > > igb driver. The old hardware is using em driver, the > CPU load is high > > but mostly due to kernel and a large pf ruleset. > > > > On the new hardware, we see high CPU Interrupt load (up > to 95%), even > > though there is not much trafic currently (peaks about > 150Mbps and > > 40Kpps). All queues are used and binded to a cpu > according to top, but a > > lot of CPU time is spent on igb queues (interrupt or > wait). The load is > > fine when we stay below 20Kpps. > > > > We see no mbuf shortage, no dropped packet, but there > is little margin > > left on CPU time (about 25% idle at best, most of CPU > time is spent on > > interrupts), which is disturbing. > > It seems you suffer from pf lock contention. You should stop > using pf > with multi-core systems with 8.3. Move to ipfw+dummynet or > ng_car for 8.3 > or move to 10.0-CURRENT having new, rewritten pf that does > not have this problem. > > Network device driver is not guilty here, that's just pf's > contention > running in igb's context. > > Eugene Grosbein They're both at play. Single threadedness aggravates subsystems that have too many lock points. It can also be "solved" with using 1 queue, because then you don't have 4 queues going into a single thread. BC ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: High CPU interrupt load on intel I350T4 with igb on 8.3
On 26.04.2013 18:31, "Clément Hermann (nodens)" wrote: > Hi list, > > We use pf+ALTQ for trafic shaping on some routers. > > We are switching to new servers : Dell PowerEdge R620 with 2 8-cores > Intel Processor (E5-2650L), 8GB RAM and Intel I350T4 (quad port) using > igb driver. The old hardware is using em driver, the CPU load is high > but mostly due to kernel and a large pf ruleset. > > On the new hardware, we see high CPU Interrupt load (up to 95%), even > though there is not much trafic currently (peaks about 150Mbps and > 40Kpps). All queues are used and binded to a cpu according to top, but a > lot of CPU time is spent on igb queues (interrupt or wait). The load is > fine when we stay below 20Kpps. > > We see no mbuf shortage, no dropped packet, but there is little margin > left on CPU time (about 25% idle at best, most of CPU time is spent on > interrupts), which is disturbing. It seems you suffer from pf lock contention. You should stop using pf with multi-core systems with 8.3. Move to ipfw+dummynet or ng_car for 8.3 or move to 10.0-CURRENT having new, rewritten pf that does not have this problem. Network device driver is not guilty here, that's just pf's contention running in igb's context. Eugene Grosbein ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: High CPU interrupt load on intel I350T4 with igb on 8.3
--- On Sun, 4/28/13, Barney Cordoba wrote: > From: Barney Cordoba > Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3 > To: "Jack Vogel" > Cc: "FreeBSD Net" , "Clément Hermann (nodens)" > > Date: Sunday, April 28, 2013, 2:59 PM > The point of lists is to be able to > benefit from other's experiences so you don't have to waste > your time "trying" things that others have already done. > I'm not pontificating. I've done the tests. There's no > reason for every person who is having to exact same problem > to do the same tests over and over, hoping for somemagically > different result. The result will always be the same. > Because there's no chance of it working properly by > chance. > BC > > > --- On Sun, 4/28/13, Jack Vogel > wrote: > > From: Jack Vogel > Subject: Re: High CPU interrupt load on intel I350T4 with > igb on 8.3 > To: "Barney Cordoba" > Cc: "FreeBSD Net" , > "Clément Hermann (nodens)" > Date: Sunday, April 28, 2013, 1:07 PM > > Try setting your queues to 1, run some tests, then try > settingyour queues to 2, then to 4... its called tuning, and > rather thanjust pontificating about it, which Barney so > loves to do, you can > discover what works best. I ran tests last week preparing > for anew driver version and found the best results came not > only whiletweaking queues, but also ring size, and I could > see changes based > on the buf ring size There are lots of things that may > improve ordegrade performance depending on the workload. > Jack > > > > On Sun, Apr 28, 2013 at 7:21 AM, Barney Cordoba > wrote: > > > > > > --- On Fri, 4/26/13, "Clément Hermann (nodens)" > wrote: > > > > > From: "Clément Hermann (nodens)" > > > Subject: High CPU interrupt load on intel I350T4 with > igb on 8.3 > > > To: freebsd-net@freebsd.org > > > Date: Friday, April 26, 2013, 7:31 AM > > > Hi list, > > > > > > We use pf+ALTQ for trafic shaping on some routers. > > >%> > > We are switching to new servers : Dell PowerEdge R620 > with 2 > > > 8-cores Intel Processor (E5-2650L), 8GB RAM and Intel > I350T4 > > > (quad port) using igb driver. The old hardware is using > em > > > driver, the CPU load is high but mostly due to kernel > and a > > > large pf ruleset. > > > > > > On the new hardware, we see high CPU Interrupt load (up > to > > > 95%), even though there is not much trafic currently > (peaks > > > about 150Mbps and 40Kpps). All queues are used and > binded to > > > a cpu according to top, but a lot of CPU time is spent > on > > > igb queues (interrupt or wait). The load is fine when > we > > > stay below 20Kpps. > > > > > > We see no mbuf shortage, no dropped packet, but there > is > > > little margin left on CPU time (about 25% idle at best, > most > > > of CPU time is spent on interrupts), which is > disturbing. > > > > > > We have done some tuning, but to no avail : > > > > > > sysctl.conf : > > > > > > # mbufs > > > kern.ipc.nmbclusters=65536 > > > # Sockets > > > kern.ipc.somaxconn=8192 > > > net.inet.tcp.delayed_ack=0 > > > net.inet.tcp.sendspace=65535 > > > net.inet.udp.recvspace=65535 > > > net.inet.udp.maxdgram=57344 > > > net.local.stream.recvspace=65535 > > > net.local.stream.sendspace=65535 > > > # IGB > > > dev.igb.0.rx_processing_limit=4096 > > > dev.igb.1.rx_processing_limit=4096 > > > dev.igb.2.rx_processing_limit=4096 > > > dev.igb.3.rx_processing_limit=4096 > > > > > > /boot/loader.conf : > > > > > > vm.kmem_size=1G > > > hw.igb.max_interrupt_rate="32000" # maximum number > of > > > interrupts/sec generated by single igb(4) (default > 8000) > > > hw.igb.txd="2048" > > > > > > # number of transmit descriptors allocated by the > > > driver (2048 limit) > > > hw.igb.rxd="2048" > > > > > > # number of receive descriptors allocated by the > > > driver (2048 limit) > > > hw.igb.rx_process_limit="1000" # > > > maximum number of received packets to process at a > time, The > > > default of 100 is > > >
Re: High CPU interrupt load on intel I350T4 with igb on 8.3
The point of lists is to be able to benefit from other's experiences so you don't have to waste your time "trying" things that others have already done. I'm not pontificating. I've done the tests. There's no reason for every person who is having to exact same problem to do the same tests over and over, hoping for somemagically different result. The result will always be the same. Because there's no chance of it working properly by chance. BC --- On Sun, 4/28/13, Jack Vogel wrote: From: Jack Vogel Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3 To: "Barney Cordoba" Cc: "FreeBSD Net" , "Clément Hermann (nodens)" Date: Sunday, April 28, 2013, 1:07 PM Try setting your queues to 1, run some tests, then try settingyour queues to 2, then to 4... its called tuning, and rather thanjust pontificating about it, which Barney so loves to do, you can discover what works best. I ran tests last week preparing for anew driver version and found the best results came not only whiletweaking queues, but also ring size, and I could see changes based on the buf ring size There are lots of things that may improve ordegrade performance depending on the workload. Jack On Sun, Apr 28, 2013 at 7:21 AM, Barney Cordoba wrote: --- On Fri, 4/26/13, "Clément Hermann (nodens)" wrote: > From: "Clément Hermann (nodens)" > Subject: High CPU interrupt load on intel I350T4 with igb on 8.3 > To: freebsd-net@freebsd.org > Date: Friday, April 26, 2013, 7:31 AM > Hi list, > > We use pf+ALTQ for trafic shaping on some routers. > > We are switching to new servers : Dell PowerEdge R620 with 2 > 8-cores Intel Processor (E5-2650L), 8GB RAM and Intel I350T4 > (quad port) using igb driver. The old hardware is using em > driver, the CPU load is high but mostly due to kernel and a > large pf ruleset. > > On the new hardware, we see high CPU Interrupt load (up to > 95%), even though there is not much trafic currently (peaks > about 150Mbps and 40Kpps). All queues are used and binded to > a cpu according to top, but a lot of CPU time is spent on > igb queues (interrupt or wait). The load is fine when we > stay below 20Kpps. > > We see no mbuf shortage, no dropped packet, but there is > little margin left on CPU time (about 25% idle at best, most > of CPU time is spent on interrupts), which is disturbing. > > We have done some tuning, but to no avail : > > sysctl.conf : > > # mbufs > kern.ipc.nmbclusters=65536 > # Sockets > kern.ipc.somaxconn=8192 > net.inet.tcp.delayed_ack=0 > net.inet.tcp.sendspace=65535 > net.inet.udp.recvspace=65535 > net.inet.udp.maxdgram=57344 > net.local.stream.recvspace=65535 > net.local.stream.sendspace=65535 > # IGB > dev.igb.0.rx_processing_limit=4096 > dev.igb.1.rx_processing_limit=4096 > dev.igb.2.rx_processing_limit=4096 > dev.igb.3.rx_processing_limit=4096 > > /boot/loader.conf : > > vm.kmem_size=1G > hw.igb.max_interrupt_rate="32000" # maximum number of > interrupts/sec generated by single igb(4) (default 8000) > hw.igb.txd="2048" > > # number of transmit descriptors allocated by the > driver (2048 limit) > hw.igb.rxd="2048" > > # number of receive descriptors allocated by the > driver (2048 limit) > hw.igb.rx_process_limit="1000" # > maximum number of received packets to process at a time, The > default of 100 is > > > > > # too low for most firewalls. (-1 means > unlimited) > > Kernel HZ is 1000. > > The IGB /boot/loader.conf tuning was our last attempt, it > didn't change anything. > > Does anyone have any pointer ? How could we lower CPU > interrupt load ? should we set hw.igb.max_interrupt_rate > lower instead of higher ? > From what we saw here and there, we should be able to do > much better with this hardware. > > > relevant sysctl (igb1 and igb2 only, other interfaces are > unused) : > > sysctl dev.igb | grep -v ": 0$" > dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection > version - 2.3.1 > dev.igb.1.%driver: igb > dev.igb.1.%location: slot=0 function=1 > dev.igb.1.%pnpinfo: vendor=0x8086 device=0x1521 > subvendor=0x8086 subdevice=0x5001 class=0x02 > dev.igb.1.%parent: pci5 > dev.igb.1.nvm: -1 > dev.igb.1.enable_aim: 1 > dev.igb.1.fc: 3 > dev.igb.1.rx_processing_limit: 4096 > dev.igb.1.eee_disabled: 1 > dev.igb.1.link_irq: 2 > dev.igb.1.device_control: 1209795137 > dev.igb.1.rx_control: 67141658
Re: High CPU interrupt load on intel I350T4 with igb on 8.3
Try setting your queues to 1, run some tests, then try setting your queues to 2, then to 4... its called tuning, and rather than just pontificating about it, which Barney so loves to do, you can discover what works best. I ran tests last week preparing for a new driver version and found the best results came not only while tweaking queues, but also ring size, and I could see changes based on the buf ring size There are lots of things that may improve or degrade performance depending on the workload. Jack On Sun, Apr 28, 2013 at 7:21 AM, Barney Cordoba wrote: > > > --- On Fri, 4/26/13, "Clément Hermann (nodens)" > wrote: > > > From: "Clément Hermann (nodens)" > > Subject: High CPU interrupt load on intel I350T4 with igb on 8.3 > > To: freebsd-net@freebsd.org > > Date: Friday, April 26, 2013, 7:31 AM > > Hi list, > > > > We use pf+ALTQ for trafic shaping on some routers. > > > > We are switching to new servers : Dell PowerEdge R620 with 2 > > 8-cores Intel Processor (E5-2650L), 8GB RAM and Intel I350T4 > > (quad port) using igb driver. The old hardware is using em > > driver, the CPU load is high but mostly due to kernel and a > > large pf ruleset. > > > > On the new hardware, we see high CPU Interrupt load (up to > > 95%), even though there is not much trafic currently (peaks > > about 150Mbps and 40Kpps). All queues are used and binded to > > a cpu according to top, but a lot of CPU time is spent on > > igb queues (interrupt or wait). The load is fine when we > > stay below 20Kpps. > > > > We see no mbuf shortage, no dropped packet, but there is > > little margin left on CPU time (about 25% idle at best, most > > of CPU time is spent on interrupts), which is disturbing. > > > > We have done some tuning, but to no avail : > > > > sysctl.conf : > > > > # mbufs > > kern.ipc.nmbclusters=65536 > > # Sockets > > kern.ipc.somaxconn=8192 > > net.inet.tcp.delayed_ack=0 > > net.inet.tcp.sendspace=65535 > > net.inet.udp.recvspace=65535 > > net.inet.udp.maxdgram=57344 > > net.local.stream.recvspace=65535 > > net.local.stream.sendspace=65535 > > # IGB > > dev.igb.0.rx_processing_limit=4096 > > dev.igb.1.rx_processing_limit=4096 > > dev.igb.2.rx_processing_limit=4096 > > dev.igb.3.rx_processing_limit=4096 > > > > /boot/loader.conf : > > > > vm.kmem_size=1G > > hw.igb.max_interrupt_rate="32000" # maximum number of > > interrupts/sec generated by single igb(4) (default 8000) > > hw.igb.txd="2048" > > > > # number of transmit descriptors allocated by the > > driver (2048 limit) > > hw.igb.rxd="2048" > > > > # number of receive descriptors allocated by the > > driver (2048 limit) > > hw.igb.rx_process_limit="1000" # > > maximum number of received packets to process at a time, The > > default of 100 is > > > > > > > > > ># too low for most firewalls. (-1 means > > unlimited) > > > > Kernel HZ is 1000. > > > > The IGB /boot/loader.conf tuning was our last attempt, it > > didn't change anything. > > > > Does anyone have any pointer ? How could we lower CPU > > interrupt load ? should we set hw.igb.max_interrupt_rate > > lower instead of higher ? > > From what we saw here and there, we should be able to do > > much better with this hardware. > > > > > > relevant sysctl (igb1 and igb2 only, other interfaces are > > unused) : > > > > sysctl dev.igb | grep -v ": 0$" > > dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection > > version - 2.3.1 > > dev.igb.1.%driver: igb > > dev.igb.1.%location: slot=0 function=1 > > dev.igb.1.%pnpinfo: vendor=0x8086 device=0x1521 > > subvendor=0x8086 subdevice=0x5001 class=0x02 > > dev.igb.1.%parent: pci5 > > dev.igb.1.nvm: -1 > > dev.igb.1.enable_aim: 1 > > dev.igb.1.fc: 3 > > dev.igb.1.rx_processing_limit: 4096 > > dev.igb.1.eee_disabled: 1 > > dev.igb.1.link_irq: 2 > > dev.igb.1.device_control: 1209795137 > > dev.igb.1.rx_control: 67141658 > > dev.igb.1.interrupt_mask: 4 > > dev.igb.1.extended_int_mask: 2147483981 > > dev.igb.1.fc_high_water: 33168 > > dev.igb.1.fc_low_water: 33152 > > dev.igb.1.queue0.interrupt_rate: 71428 > > dev.igb.1.queue0.txd_head: 1318 > > dev.igb.1.queue0.txd_tail: 1318 > > dev.igb.1.queue0.tx_packets: 84663594 > > dev.igb.1.queue0.rxd_head: 717 > > dev.igb.1.queue0.rxd_tail: 715 > > dev.igb.1.queue0.rx_packets: 43899597 > > dev.igb.1.queue0.rx_bytes: 8905556030 > > dev.igb.1.queue1.interrupt_rate: 90909 > > dev.igb.1.queue1.txd_head: 693 > > dev.igb.1.queue1.txd_tail: 693 > > dev.igb.1.queue1.tx_packets: 57543349 > > dev.igb.1.queue1.rxd_head: 1033 > > dev.igb.1.queue1.rxd_tail: 1032 > > dev.igb.1.queue1.rx_packets: 54821897 > > dev.igb.1.queue1.rx_bytes: 9944955108 > > dev.igb.1.queue2.interrupt_rate: 10 > > dev.igb.1.queue2.txd_head: 350 > > dev.igb.1.queue2.txd_tail: 350 > > dev.igb.1.queue2.tx_packets: 62320990 > > dev.igb.1.queue2.rxd_head: 1962 > > dev.igb.1.queue2.rxd_tail: 1939 > > dev.igb.1.queue2.rx_packets: 43909016 > > dev.igb.1.queue2.rx_bytes: 8673941461 > > dev.igb.1.queue3.interrupt_rate: 14925 > > dev.igb.1.queue3.txd_he
Re: High CPU interrupt load on intel I350T4 with igb on 8.3
--- On Fri, 4/26/13, "Clément Hermann (nodens)" wrote: > From: "Clément Hermann (nodens)" > Subject: High CPU interrupt load on intel I350T4 with igb on 8.3 > To: freebsd-net@freebsd.org > Date: Friday, April 26, 2013, 7:31 AM > Hi list, > > We use pf+ALTQ for trafic shaping on some routers. > > We are switching to new servers : Dell PowerEdge R620 with 2 > 8-cores Intel Processor (E5-2650L), 8GB RAM and Intel I350T4 > (quad port) using igb driver. The old hardware is using em > driver, the CPU load is high but mostly due to kernel and a > large pf ruleset. > > On the new hardware, we see high CPU Interrupt load (up to > 95%), even though there is not much trafic currently (peaks > about 150Mbps and 40Kpps). All queues are used and binded to > a cpu according to top, but a lot of CPU time is spent on > igb queues (interrupt or wait). The load is fine when we > stay below 20Kpps. > > We see no mbuf shortage, no dropped packet, but there is > little margin left on CPU time (about 25% idle at best, most > of CPU time is spent on interrupts), which is disturbing. > > We have done some tuning, but to no avail : > > sysctl.conf : > > # mbufs > kern.ipc.nmbclusters=65536 > # Sockets > kern.ipc.somaxconn=8192 > net.inet.tcp.delayed_ack=0 > net.inet.tcp.sendspace=65535 > net.inet.udp.recvspace=65535 > net.inet.udp.maxdgram=57344 > net.local.stream.recvspace=65535 > net.local.stream.sendspace=65535 > # IGB > dev.igb.0.rx_processing_limit=4096 > dev.igb.1.rx_processing_limit=4096 > dev.igb.2.rx_processing_limit=4096 > dev.igb.3.rx_processing_limit=4096 > > /boot/loader.conf : > > vm.kmem_size=1G > hw.igb.max_interrupt_rate="32000" # maximum number of > interrupts/sec generated by single igb(4) (default 8000) > hw.igb.txd="2048" > > # number of transmit descriptors allocated by the > driver (2048 limit) > hw.igb.rxd="2048" > > # number of receive descriptors allocated by the > driver (2048 limit) > hw.igb.rx_process_limit="1000" # > maximum number of received packets to process at a time, The > default of 100 is > > > > > # too low for most firewalls. (-1 means > unlimited) > > Kernel HZ is 1000. > > The IGB /boot/loader.conf tuning was our last attempt, it > didn't change anything. > > Does anyone have any pointer ? How could we lower CPU > interrupt load ? should we set hw.igb.max_interrupt_rate > lower instead of higher ? > From what we saw here and there, we should be able to do > much better with this hardware. > > > relevant sysctl (igb1 and igb2 only, other interfaces are > unused) : > > sysctl dev.igb | grep -v ": 0$" > dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection > version - 2.3.1 > dev.igb.1.%driver: igb > dev.igb.1.%location: slot=0 function=1 > dev.igb.1.%pnpinfo: vendor=0x8086 device=0x1521 > subvendor=0x8086 subdevice=0x5001 class=0x02 > dev.igb.1.%parent: pci5 > dev.igb.1.nvm: -1 > dev.igb.1.enable_aim: 1 > dev.igb.1.fc: 3 > dev.igb.1.rx_processing_limit: 4096 > dev.igb.1.eee_disabled: 1 > dev.igb.1.link_irq: 2 > dev.igb.1.device_control: 1209795137 > dev.igb.1.rx_control: 67141658 > dev.igb.1.interrupt_mask: 4 > dev.igb.1.extended_int_mask: 2147483981 > dev.igb.1.fc_high_water: 33168 > dev.igb.1.fc_low_water: 33152 > dev.igb.1.queue0.interrupt_rate: 71428 > dev.igb.1.queue0.txd_head: 1318 > dev.igb.1.queue0.txd_tail: 1318 > dev.igb.1.queue0.tx_packets: 84663594 > dev.igb.1.queue0.rxd_head: 717 > dev.igb.1.queue0.rxd_tail: 715 > dev.igb.1.queue0.rx_packets: 43899597 > dev.igb.1.queue0.rx_bytes: 8905556030 > dev.igb.1.queue1.interrupt_rate: 90909 > dev.igb.1.queue1.txd_head: 693 > dev.igb.1.queue1.txd_tail: 693 > dev.igb.1.queue1.tx_packets: 57543349 > dev.igb.1.queue1.rxd_head: 1033 > dev.igb.1.queue1.rxd_tail: 1032 > dev.igb.1.queue1.rx_packets: 54821897 > dev.igb.1.queue1.rx_bytes: 9944955108 > dev.igb.1.queue2.interrupt_rate: 10 > dev.igb.1.queue2.txd_head: 350 > dev.igb.1.queue2.txd_tail: 350 > dev.igb.1.queue2.tx_packets: 62320990 > dev.igb.1.queue2.rxd_head: 1962 > dev.igb.1.queue2.rxd_tail: 1939 > dev.igb.1.queue2.rx_packets: 43909016 > dev.igb.1.queue2.rx_bytes: 8673941461 > dev.igb.1.queue3.interrupt_rate: 14925 > dev.igb.1.queue3.txd_head: 647 > dev.igb.1.queue3.txd_tail: 647 > dev.igb.1.queue3.tx_packets: 58776199 > dev.igb.1.queue3.rxd_head: 692 > dev.igb.1.queue3.rxd_tail: 691 > dev.igb.1.queue3.rx_packets: 55138996 > dev.igb.1.queue3.rx_bytes: 9310217354 > dev.igb.1.queue4.interrupt_rate: 10 > dev.igb.1.queue4.txd_head: 1721 > dev.igb.1.queue4.txd_tail: 1721 > dev.igb.1.queue4.tx_packets: 54337209 > dev.igb.1.queue4.rxd_head: 1609 > dev.igb.1.queue4.rxd_tail: 1598 > dev.igb.1.queue4.rx_packets: 46546503 > dev.igb.1.queue4.rx_bytes: 8818182840 > dev.igb.1.queue5.interrupt_rate: 11627 > dev.igb.1.queue5.txd_head: 254 > dev.igb.1.queue5.txd_tail: 254 > dev.igb.1.queue5.tx_packets: 53117182 > dev.igb.1.queue5.rxd_head: 701 > dev.
