2009/3/3 Brandeburg, Jesse <[email protected]>: > Matthew Faulkner wrote: >> I have two Intel 10 Gigabit CX4 Dual Port Server Adaptors, which I >> believe use the 82598EB chipset. I've put these cards in two machines >> and connected them point-to-point with the CX4 cable. > > okay > >> I'm measuring the performance between these two machines. >> I'm using Jumbo frames (9000MTU) and both ports. >> The machines are both multi core machines with affinities set so that >> the CPU is not the limiting the speed. > > okay, good > >> However, the absolute maximum speed I can achieve is ~1400MB/s, with >> each port achieving equal bandwidth (~700MB/s). These machines are >> also connected by two 1 gig cards. When i include these cards in the >> performance test, the maximum bandwidth increases up to ~1645MB/s >> (with both 1gig cards achieving ~123MB/s). So it seems doubtful to me >> that the bus is limiting the speed. > > transmitting, receiving or both?
The numbers quoted here is the application goodput as seen on the receiving side. > >> My question is what is limiting it? Am i misunderstanding the CX4 card >> by thinking it's two 10GbE which would give me a theoretical maximum >> of 2500MB/s. Does the card infact only give me a total of 10GB/S >> (1250MB/s)? Is so then how do the two ports go faster than 10GbE? > > each port is capable at the wire of 20 Gb/s (10 Gb/s full duplex) > aka each port is capable of 1.2GB/s (or 1200MB/s) in each direction. Thats > the easy part. > >> Or is there something I'm missing entirely? > > maybe a little. > >> I'll be grateful of any help you can provide. If you require the >> settings of the driver i'm using (ixgbe) or the outputs of lspci, >> dmesg etc please email. I've not included them in this email to keep >> it short :) > > As it turns out a x8 PCIe slot is only capable of 40 Gb/s raw, and that is > 8/10 bit encoded so that takes it down to 32Gb/s absolute maximum data > payload rate (including all configuration transactions, etc.) > this is for the slot only (so two ports share it) > > If you use two cards (using a single port each one) you'll probably see > higher numbers. > > The increases you see with 1Gb/s cards implies that your system is not > memory bandwidth limited. > > on a single port you should be able to see line rate bidirectional, the x8 > PCIe Gen 1 slot limits dual port throughput a bit. > > Hope this helps, > Jesse > So these numbers suggest that I should be able to get above the 1400MB/s (just above 11Gb/s) that i am getting? ------------------------------------------------------------------------------ Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise -Strategies to boost innovation and cut costs with open source participation -Receive a $600 discount off the registration fee with the source code: SFAD http://p.sf.net/sfu/XcvMzF8H _______________________________________________ E1000-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/e1000-devel
