Yo, LRO is an interesting hack that seems to do a good trick of hiding the ridiculous locking and unfriendly cache behaviour that we do per-packet.
It helps with LAN test traffic where things are going out in batches from the TCP layer so the RX layer "sees" these frames in-order and can do LRO. When you disable it, I don't easily get 10GE LAN TCP performance. That has to be fixed. Given how fast the CPU cores, bus interconnect and memory interconnects are, I don't think there should be any reason why we can't hit 10GE traffic on a LAN with LRO disabled (in both software and hardware.) Now that I have the PMC sandy bridge stuff working right (but no PEBS, I have to talk to Intel about that in a bit more detail before I think about hacking that in) we can get actual live information about this stuff. But the last time I looked, there's just too much per-packet latency going on. The root cause looks like it's a toss up between scheduling, locking and just lots of code running to completion per-frame. As I said, that all has to die somehow. 2c, -adrian On 1 September 2013 08:45, Barney Cordoba <barney_cord...@yahoo.com> wrote: > > > Comcast sends packets OOO. With any decent number of internet hops you're > likely to encounter a load > balancer or packet shaper that sends packets OOO, so you just can't be > worried about it. In fact, your > designs MUST work with OOO packets. > > Getting balance on your load balanced lines is certainly a bigger upside > than the additional CPU used. > You can buy a faster processor for your "stack" for a lot less than you > can buy bandwidth. > > Frankly my opinion of LRO is that it's a science project suitable for labs > only. It's a trick to get more bandwidth > than your bus capacity; the answer is to not run PCIe2 if you need pcie3. > You can use it internally if you have > control of all of the machines. When I modify a driver the first thing > that I do is rip it out. > > BC > > > ________________________________ > From: Luigi Rizzo <ri...@iet.unipi.it> > To: Barney Cordoba <barney_cord...@yahoo.com> > Cc: Andre Oppermann <an...@freebsd.org>; Alan Somers <asom...@freebsd.org>; > "n...@freebsd.org" <n...@freebsd.org>; Jack F Vogel <j...@freebsd.org>; > Justin T. Gibbs <gi...@freebsd.org>; T.C. Gubatayao < > tgubata...@barracuda.com> > Sent: Saturday, August 31, 2013 10:27 PM > Subject: Re: Flow ID, LACP, and igb > > > On Sun, Sep 1, 2013 at 4:15 AM, Barney Cordoba <barney_cord...@yahoo.com > >wrote: > > > ... > > > > [your point on testing with realistic assumptions is surely a valid one] > > > > > > Of course there's nothing really wrong with OOO packets. We had this > > discussion before; lots of people > > have round robin dual homing without any ill effects. It's just not an > > issue. > > > > It depends on where you are. > It may not be an issue if the reordering is not large enough to > trigger retransmissions, but even then it is annoying as it causes > more work in the endpoint -- it prevents LRO from working, and even > on the host stack it takes more work to sort where an out of order > segment goes than appending an in-order one to the socket buffer. > > cheers > luigi > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" > _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
