I am trying to properly setup/understand RSS and smp_affinity using the scripts
(load_driver and set_irq_affinity) provided with the PF_RING tarball without
CPU binding. I will put the questions upfront and the details will follow. I
would appreciate any help, though the questions may not be PF_RING tightly
related. Other seupt/configurations are already done and are working as
expected.
1. How do the values assigned to RSS when insmod(ing) the driver actually map
to the queues? Such as RSS=0,0 or RS=1,1,1,1 or RSS=4,4.2. The application at
the other end that is consuming the traffic has 4 processes running. Does this
mean that only 4 queues/4 CPUs should be mapped, or I can have the max. number
of queues irq(ed) - in this case 8 - to the 6 cores and let the scheduler
handle the reset?3. What is the easiest way for backing-out if things do not go
well; would rmmod(ing) and reloading the driver suffice? This is because I am
dealing with a production system already and don't have the space to test.
Loading igb and pf_ring:
insmod ./pf_ring.ko transparent_mode=2 min_num_slots=65536
enable_tx_capture=0insmod ./igb.ko
Server info:
# cat /proc/cpuinfo | grep "model name\|cpu cores" | head -2model name :
Intel(R) Xeon(R) CPU E5-2420 v2 @ 2.20GHzcpu cores : 6
# uname -r3.2.0-67-generic
PF_RING info:
# cat /proc/net/pf_ring/infoPF_RING Version : 6.0.2 ($Revision:
8089$)Total rings : 4
Standard (non DNA) OptionsRing slots : 65536Slot version
: 16Capture TX : No [RX only]IP Defragment
: NoSocket Mode : StandardTransparent mode
: No [mode 2]Total plugins : 0Cluster Fragment Queue :
0Cluster Fragment Discard : 0
Driver info:
# ethtool -i eth2driver: igbversion: 5.2.5firmware-version: 1.67, 0x80000ba6,
15.0.28bus-info: 0000:08:00.0supports-statistics: yessupports-test:
yessupports-eeprom-access: yessupports-register-dump: yes
Looking at dmesg for the driver suggest that there are 8 RX queues and 8 TX
queues:
# grep "igb" /var/log/dmesg[ 3.167209] igb 0000:08:00.0: Intel(R) Gigabit
Ethernet Network Connection[ 3.167214] igb 0000:08:00.0: eth2:
(PCIe:5.0Gb/s:Width x4) <MAC-ADDRESS>[ 3.167511] igb 0000:08:00.0: eth2: PBA
No: G13158-000[ 3.167513] igb 0000:08:00.0: Using MSI-X interrupts. 8 rx
queue(s), 8 tx queue(s)
However, listing the queues associated, does not show that are actually being
assigned/used;
# ls -l /sys/class/net/eth2/queues/total 0drwxr-xr-x 2 root root 0 Aug 28 08:56
rx-0drwxr-xr-x 2 root root 0 Aug 28 08:56 tx-0
And:
# cat /proc/interrupts | grep "eth2" CPU0 CPU1
CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8
CPU9 CPU10 CPU11 81: 1 0
0 0 0 0 0 0
0 0 0 0 IR-PCI-MSI-edge eth2
82: 1857321564 0 0 0 0
0 0 0 0 0 0
0 IR-PCI-MSI-edge eth2-TxRx-0
If I am reading this correctly, then there is only one queue and it is being
handled by CPU0 only and the other cores are just dormant. What is also
confusing me is:
# cat /proc/irq/82/smp_affinity0000,00000fff
Again, if I am reading this correctly, then this means that IRQ 82 is (should
be) handled by CPU0...CPU11, however, this is not evident in the interrupt
table above. Looking at the indirection table also gives nothing:
# ethtool -x eth2RX flow hash indirection table for eth2 with 1 RX ring(s):
0: 0 0 0 0 0 0 0 0 8: 0 0 0
0 0 0 0 0 16: 0 0 0 0 0 0 0
0 24: 0 0 0 0 0 0 0 0 32: 0 0
0 0 0 0 0 0 40: 0 0 0 0 0 0
0 0 48: 0 0 0 0 0 0 0 0 56: 0
0 0 0 0 0 0 0 64: 0 0 0 0 0
0 0 0 72: 0 0 0 0 0 0 0 0 80:
0 0 0 0 0 0 0 0 88: 0 0 0 0
0 0 0 0 96: 0 0 0 0 0 0 0 0
104: 0 0 0 0 0 0 0 0 112: 0 0 0
0 0 0 0 0 120: 0 0 0 0 0 0 0
0
And finally:
# ethtool -S eth2NIC statistics: rx_packets: 2984011999 tx_packets: 3
rx_bytes: 2465341976853 tx_bytes: 222 rx_broadcast: 0
tx_broadcast: 0 rx_multicast: 0 tx_multicast: 3 multicast: 0
collisions: 0 rx_crc_errors: 0 rx_no_buffer_count: 0
rx_missed_errors: 0 tx_aborted_errors: 0 tx_carrier_errors: 0
tx_window_errors: 0 tx_abort_late_coll: 0 tx_deferred_ok: 0
tx_single_coll_ok: 0 tx_multi_coll_ok: 0 tx_timeout_count: 0
rx_long_length_errors: 0 rx_short_length_errors: 0 rx_align_errors: 0
tx_tcp_seg_good: 0 tx_tcp_seg_failed: 0 rx_flow_control_xon: 0
rx_flow_control_xoff: 0 tx_flow_control_xon: 0 tx_flow_control_xoff: 0
rx_long_byte_count: 2465341976853 tx_dma_out_of_sync: 0
lro_aggregated: 0 lro_flushed: 0 tx_smbus: 0 rx_smbus: 0
dropped_smbus: 0 os2bmc_rx_by_bmc: 0 os2bmc_tx_by_bmc: 0
os2bmc_tx_by_host: 0 os2bmc_rx_by_host: 0 rx_errors: 0 tx_errors: 0
tx_dropped: 0 rx_length_errors: 0 rx_over_errors: 0
rx_frame_errors: 0 rx_fifo_errors: 0 tx_fifo_errors: 0
tx_heartbeat_errors: 0 tx_queue_0_packets: 3 tx_queue_0_bytes: 210
tx_queue_0_restart: 0 rx_queue_0_packets: 2984012000 rx_queue_0_bytes:
2453405930211 rx_queue_0_drops: 0 rx_queue_0_csum_err: 0
rx_queue_0_alloc_failed: 0
Again, I appreciate any help. Thanks. (Sorry if this is a duplicate).YM
_______________________________________________
Ntop-misc mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop-misc
