Hello, about a month ago I wrote I'm glad about em(4) driver which works pretty well on few of my boxes. However I need to change my opinion... after what I saw today in the lab:
We have connected pretty well testing box - Navtel InterWatch (www.navtelcom.com). It has one 6 slots and in one of them it has 2 GigE TX ports. We've configured the following scenario: 1) Navtel port A --- em0 --- em1 --- Navtel port B 2) Navtel port A --- em2 --- em3 --- Navtel port B em0 and em1 are built-in mainboard: em0 at pci4 dev 0 function 0 "Intel PRO/1000 PT (82571EB)" rev 0x06: apic 2 int 16 (irq 10), em1 at pci4 dev 0 function 1 "Intel PRO/1000 PT (82571EB)" rev 0x06: apic 2 int 17 (irq 11), and em2 and em3 are Intel Dual Port Server Adapter on PCI-e (4x): em2 at pci5 dev 0 function 0 "Intel PRO/1000MT (82573E)" rev 0x03: apic 2 int 19 (irq 10), em3 at pci6 dev 0 function 0 "Intel PRO/1000MT (82573L)" rev 0x00: apic 2 int 16 (irq 11), pf is disabled, between em0 and em1 whole traffic goes through kernel routing process (Navtel port A and em0 in one /24 network, and em1 and Navtel port B are in different /24 network) sysctl tcp.send & receive space is turned to 65535 When we generate 500Mbps of traffic (1434 bytes in ethernet, which gives 1400 bytes of payload in TCP) from port A to B and from B to A (two streams, each of 500Mbps) everything works pretty well on PT chips and MT chips. When we change payload to 64bytes (called IP killer :P) and put it in scenario 1) machine gets freeze and no packet is comming to port B of Navtel device. It's rather normal, there are no ASIC-based boxes which can work with such traffic :) What was strange, when we connected Navtel to em2 and em3 which is one PCI-e (4x) dual-port card and started to generate traffic from port A to B and from B to A machine has restarted about a second after test started. We've change sysctl values to move machine to debugger if anything goes bad, but it didn't change anything. I think that it's somehow connected to chipsets of that cards and mainboard bridges which are responsible for transferring packets through the mainboard. Anyway, I started to feel badly about Intel... Second test was to generate 900Mbps of pure IP traffice (payload 1400 bytes) from port A to B and second stream from B to A. In scenario 1 machine got freeze and hasn't forward any packet from em0 to em1. When we changed em0/1 to em2/3 all traffic is comming from port A to B without any loss and machine gets 75% interrupt on uniprocessor kernel. So, we've changed to MP kernel and... scenario 1) hasn't changed at all (freeze all the time), and scenario 2 got 100% idle and 0% interrupts on both CPUs (strange, I thought that it'll be 1/2 of previous 75% :P). Anyway, when we connected anything to em0/1 ports during that test and generate more than 100Mbps (bittwist software packet generator run at the second box) our test machine got freeze again... What else? Kernel is taken from CVS -current tree. After all these tests I'm changing my opinion about Intel cards, especially when I read that PT chipsets are Intel's newest "baby". Does anyone got simmilar problems ? Maybe there are other ways to tune NICs to work under such traffic (buffers on NIC?). I'm not an expert in Intel network cards so any idea will be appreciated :) Maybe you can tell about other chipsets that works fine under such "heavy" traffic ? -- regards, Sylwester S. Biernacki <[EMAIL PROTECTED]> X-NET, http://www.xnet.com.pl/
