On Thu, Aug 25, 2011 at 10:43 PM, George Porter <[email protected]> wrote: > Hi Mihai, > > On Thu, Aug 25, 2011 at 10:16 AM, Mihai Dobrescu > <[email protected]> wrote: >> For the moment I don't have a proper setup to test the described >> behavior, but the problem seems to be related to the batch settings in >> the NIC driver. >> >> b1) An easy fix would be to try to change the the batch value from 16 >> to 8 or 4 to see if the latency improves. Check the driver's source >> folder (ixgbe_main.c) and look for something like rxdctl |= (16<<16); >> txdctl |= (16<<16). > > I tried adjusting this parameter to a few different settings. It > worked with batching levels of 16 and 8. For some reason when I tried > 4 and 2 packets we're delivered out of the driver regardless of how > many I sent. > > Looking at the latency, it seems that for small bursts of packets > (e.g., 8, 16, 32 and thereabouts) the latency through RouteBricks is > very low--about 14us or so.
That's good news :) > > When I try large packet bursts (e.g., 16,000 packets back to back) > there is an interesting queueing-type behavior in which the latency > for the earlier packets is 14us, and then it linearly climbs up to > ~560us or so, and then jumps back down to 14 and repeats in a sawtooth > pattern. There are 2 forms of batching: NIC batching and Click batching. The described behavior might be related to Click batching. You could check the "BURST" parameters in PollDevice and ToDevice to adjust the Click batching. > This may be due to an RSS hash issue, given that the receive > queue stats show that most packets end up in a single queue (vs being > spread over the 4 queues that are instantiated in the nic). I'm going > to have my packet generator randomize the source IP address to > hopefully spread those packets over those queues and retry. > The RSS delivery to a particular queue is done based on the (SrcIP,DstIP,SrcPort,DstPort) tuple. If you send packets using the same headers, they will all end up in the same queue. Mihai _______________________________________________ click mailing list [email protected] https://amsterdam.lcs.mit.edu/mailman/listinfo/click
