Hi everyone,
I’m new to PF_RING so I’m no doubt doing something silly :) I’ve looked
through the ntop-misc archives as far back as Jan 2012, and as far as I
can see my particular circumstances haven’t been raised before.
I’ll describe my setup first, then the actual problem.
I’m studying the PF_RING driver and userland code to try and progress
this myself, but I’d welcome any suggestions for things to check or try.
Regards,
A.
THE HARDWARE
Two HP ProLiant DL360 G6 machines, each with 2 x 4-core 2.66GHz X5550
Xeons, plus PCIe v2 and 12Gb DDR3 RAM.
Machine 1 has a Napatech NT20-E2 card.
Machine 2 has an 82599-based dual-port HP560FLR card in an 8-lane slot.
Hyperthreading has been disabled on this machine, giving 8 cores.
Machine 1 interface napa0 is patched directly to machine 2 eth3. For
the purposes of this discussion, I’m using machine1/Napatech as a packet
generator to test PF_RING on machine 2/Intel 82599.
THE SOFTWARE
Both machines running RHEL 6.6 – uname says 2.6.32-504.23.4.el6.x86_64.
On machine 2, PF_RING is v6.0.3, and I compiled with:-
cd PF_RING-6.0.3
make
cd drivers
make
I have NOT performed a ‘make install’ anywhere in the PF_RING directory
tree (I wasn’t sure what ‘make install’ would do to my existing libpcap
1.7.4 installation. I run multiple SSH logins, each with
C_INCLUDE_PATH and LD_LIBRARY_PATH set accordingly. Within one login my
code will build and link to PF_RING libs; in another login it will build
and link to standard libpcap).
The pfcount and pf_ring.ko/ixgbe.ko executables are run directly from
within this tree.
Drivers are started via:-
cd drivers/ZC/intel/ixgbe/ixgbe-3.22.3-zc/src
./load_driver.sh
and I’ve experimented with various driver options for insmod pf_ring and
insmod ixgbe (and confirmed in dmesg that the options have been accepted).
The example app is then run via:-
../../../../../../userland/examples/pfcount -i eth3
or
../../../../../../userland/examples/pfcount -i zc:eth3
The ZC variant issues a warning about no ZC licence / running in demo
mode, which is fine for my testing.
THE PROBLEM
I’m testing PF_RING’s ability to receive sustained bursts of
9-10Gbits/sec traffic.
I have a 110 Mb traffic recording in standard pcap dump file format. On
machine 1, I’m using my own utility to read the dump file and inject
each packet to interface napa0 as fast as possible. (The entire file is
sent in < 1 second. I also have a 10Gb recording, and the Napatech
‘monitor’ utility confirms that the napa0 TX data rate (including
preamble and IFG) is 9999-10000 Gbps.)
When sending the 110Mb file, pfcount –i eth3 reports approx. 38% dropped
packets, and approx. 65Mbs data received. This is fine, I would not
expect 100% data reception. As a test I can throttle back TX and send
packets with a short nanosecond sleep between each packet, and pfcount
reports 100% reception.
When running with pfcount –i zc:eth3, the following stats are
consistently reported:-
Packets received – 22335 (somewhat less than the 178K packets transmitted)
Packets dropped – 0
Data received – approx 13Mbs (somewhat less than the 110Mb source file).
These same stats are always reported, regardless of whether TX is at
full rate or throttled back.
I have experimented with various options to the ixgbe driver, and I have
thus far not succeeded in 100% packet reception in ZC mode. I have tried:-
insmod ./ixgbe.ko (i.e. take all defaults)
insmod ./ixgbe.ko RSS=0,0
insmod ./ixgbe.ko DCA=1,1
insmod ./ixgbe.ko MQ=1,1
plus various combinations of the RSS, DCA and MQ options on one command
line.
I’m currently running with all defaults on the pf_ring driver.
I’m allowing queue-to-core affinity to be in set in
../scripts/set_irq_affinity as supplied – no script modifications to
ethtool settings or core affinity.
Finally, irqbalance is not running on my machinery.
--
Andrew Howard
[email protected]
_______________________________________________
Ntop-misc mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop-misc
