You have to admit there's a problem before you can fix it. If Eugene is going to blame to bottleneck and no one is going to tell him he's wrong, then there is no discussion.
The solution in this case is to use 1 queue, which was my suggestion many days ago. The defaults are broken. The driver should default to 1 queue, and be tuned to the system environment. With 2 NICs in the box, the defaults are defective. 1 queue should always work. Other settings require tuning and an understanding of how things work. I've had to support i350 so I've been playing with the driver a bit. It works fine with lots of cores. But you have to have more cores than queues. 2 cards with 4 queues on a 6 physical core system gets into a contention problem at certain loads. I've also removed the cpu bindings, which is about all I'm free to disclose. The driver needs a tuning doc as much as anything else. BC --- On Sat, 5/11/13, Adrian Chadd <adr...@freebsd.org> wrote: > From: Adrian Chadd <adr...@freebsd.org> > Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3 > To: "Hooman Fazaeli" <hoomanfaza...@gmail.com> > Cc: "Barney Cordoba" <barney_cord...@yahoo.com>, ""Clément Hermann (nodens)"" > <nodens2...@gmail.com>, "Eugene Grosbein" <egrosb...@rdtc.ru>, > freebsd-net@freebsd.org > Date: Saturday, May 11, 2013, 6:16 PM > Hi, > > The motivation behind the locking scheme in igb in friends > is for a > very specific, userland-traffic-origin workload. > > Sure, it may or may not work well for forwarding/filtering > workloads. > > If you want to fix it, let's have a discussion about how to > do it, > followed by some patches to do so. > > > > > Adrian > > On 11 May 2013 13:12, Hooman Fazaeli <hoomanfaza...@gmail.com> > wrote: > > On 5/11/2013 8:26 PM, Barney Cordoba wrote: > >> Clearly you don't understand the problem. Your > logic is that because other drivers are defective also; > therefore its not a driver problem? The problem is caused by > a multi-threaded driver that > >> haphazardly launches tasks and that doesn't manage > the case that the rest of the system can't handle the load. > It's no different than a driver that barfs when mbuf > clusters are exhausted. The answer > >> isn't to increase memory or mbufs, even though that > may alleviate the problem. The answer is to fix the driver, > so that it doesn't crash the system for an event that is > wholly predictable. igb has > >> 1) too many locks and 2) exasperates the problem by > binding to cpus, which causes it to not only have to wait > for the lock to free, but also for a specific cpu to become > free. So it chugs along > >> happily until it encounters a bottleneck, at which > point it quickly blows up the entire system in a domino > effect. It needs to manage locks more efficiently, and also > to detect when the backup is > >> unmanageable. Ever since FreeBSD 5 the answer has > been "it's fixed in 7, or its fixed in 9, or it's fixed in > 10". There will always be bottlenecks, and no driver should > blow up the system no matter > >> what intermediate code may present a problem. Its > the driver's responsibility to behave and to drop packets if > necessary. BC > > > > And how the driver should behave? You suggest dropping > the packets. Even if we accept > > that dropping packets is a good strategy in all > configurations (which I doubt), the driver is > > definitely not the best place to implement it, since > that involves duplication of similar > > code between drivers. Somewhere like the Ethernet layer > is a much better choice to watch > > load of packets and drop them to prevent them to eat > all the cores. Furthermore, ignoring > > the fact that pf is not optimized for multi-processors > and blaming drivers for not adjusting > > themselves with the this pf's fault, is a bit unfair, I > believe. > > > > > > -- > > > > Best regards. > > Hooman Fazaeli > > > > _______________________________________________ > > freebsd-net@freebsd.org > mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" > _______________________________________________ > freebsd-net@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" > _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
