Thanks Alfredo, Ok, If I am getting this right, If I want to establish queues = # of cores, then I would go with RSS = 0 (for one physical interface) and RS = 0,0 (for two physical interfaces), and for 4 queues only RSS = 4 (for one physical interface), correct? Yes I am using the standard mode. What I meant by "things do not go well" is that after making the necessary changes - insmod the driver with RSS and setting smp_affinity - and I find that performance was degraded (mostly an error from my side :)). I am just trying to understand the process of reverting back to the original state before the changes took place. Thanks again. YM
From: [email protected] Date: Wed, 3 Sep 2014 19:28:19 +0200 To: [email protected] Subject: Re: [Ntop-misc] Advice on setting RSS/smp_affinity Hiplease read below On 03 Sep 2014, at 14:33, Y M <[email protected]> wrote:I am trying to properly setup/understand RSS and smp_affinity using the scripts (load_driver and set_irq_affinity) provided with the PF_RING tarball without CPU binding. I will put the questions upfront and the details will follow. I would appreciate any help, though the questions may not be PF_RING tightly related. Other seupt/configurations are already done and are working as expected. 1. How do the values assigned to RSS when insmod(ing) the driver actually map to the queues? Such as RSS=0,0 or RS=1,1,1,1 or RSS=4,4. Each number in the comma-separated list is per interface, where:0 - number of RSS queue = number of CPU cores1- single queue4 - 4 RSS queues 2. The application at the other end that is consuming the traffic has 4 processes running. Does this mean that only 4 queues/4 CPUs should be mapped, or I can have the max. number of queues irq(ed) - in this case 8 - to the 6 cores and let the scheduler handle the reset? If you are using DNA/ZC you should use 4 queues and open one queue per application,with standard drivers it depends, if you are using in-kernel clustering you can use as many queues as you want. 3. What is the easiest way for backing-out if things do not go well; would rmmod(ing) and reloading the driver suffice? This is because I am dealing with a production system already and don't have the space to test. What do you mean with “things do not go well”? Alfredo Loading igb and pf_ring: insmod ./pf_ring.ko transparent_mode=2 min_num_slots=65536 enable_tx_capture=0insmod ./igb.ko Server info: # cat /proc/cpuinfo | grep "model name\|cpu cores" | head -2model name : Intel(R) Xeon(R) CPU E5-2420 v2 @ 2.20GHzcpu cores : 6 # uname -r3.2.0-67-generic PF_RING info: # cat /proc/net/pf_ring/infoPF_RING Version : 6.0.2 ($Revision: 8089$)Total rings : 4 Standard (non DNA) OptionsRing slots : 65536Slot version : 16Capture TX : No [RX only]IP Defragment : NoSocket Mode : StandardTransparent mode : No [mode 2]Total plugins : 0Cluster Fragment Queue : 0Cluster Fragment Discard : 0 Driver info: # ethtool -i eth2driver: igbversion: 5.2.5firmware-version: 1.67, 0x80000ba6, 15.0.28bus-info: 0000:08:00.0supports-statistics: yessupports-test: yessupports-eeprom-access: yessupports-register-dump: yes Looking at dmesg for the driver suggest that there are 8 RX queues and 8 TX queues: # grep "igb" /var/log/dmesg[ 3.167209] igb 0000:08:00.0: Intel(R) Gigabit Ethernet Network Connection[ 3.167214] igb 0000:08:00.0: eth2: (PCIe:5.0Gb/s:Width x4) <MAC-ADDRESS>[ 3.167511] igb 0000:08:00.0: eth2: PBA No: G13158-000[ 3.167513] igb 0000:08:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) However, listing the queues associated, does not show that are actually being assigned/used; # ls -l /sys/class/net/eth2/queues/total 0drwxr-xr-x 2 root root 0 Aug 28 08:56 rx-0drwxr-xr-x 2 root root 0 Aug 28 08:56 tx-0 And: # cat /proc/interrupts | grep "eth2" CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 81: 1 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth2 82: 1857321564 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth2-TxRx-0 If I am reading this correctly, then there is only one queue and it is being handled by CPU0 only and the other cores are just dormant. What is also confusing me is: # cat /proc/irq/82/smp_affinity0000,00000fff Again, if I am reading this correctly, then this means that IRQ 82 is (should be) handled by CPU0...CPU11, however, this is not evident in the interrupt table above. Looking at the indirection table also gives nothing: # ethtool -x eth2RX flow hash indirection table for eth2 with 1 RX ring(s): 0: 0 0 0 0 0 0 0 0 8: 0 0 0 0 0 0 0 0 16: 0 0 0 0 0 0 0 0 24: 0 0 0 0 0 0 0 0 32: 0 0 0 0 0 0 0 0 40: 0 0 0 0 0 0 0 0 48: 0 0 0 0 0 0 0 0 56: 0 0 0 0 0 0 0 0 64: 0 0 0 0 0 0 0 0 72: 0 0 0 0 0 0 0 0 80: 0 0 0 0 0 0 0 0 88: 0 0 0 0 0 0 0 0 96: 0 0 0 0 0 0 0 0 104: 0 0 0 0 0 0 0 0 112: 0 0 0 0 0 0 0 0 120: 0 0 0 0 0 0 0 0 And finally: # ethtool -S eth2NIC statistics: rx_packets: 2984011999 tx_packets: 3 rx_bytes: 2465341976853 tx_bytes: 222 rx_broadcast: 0 tx_broadcast: 0 rx_multicast: 0 tx_multicast: 3 multicast: 0 collisions: 0 rx_crc_errors: 0 rx_no_buffer_count: 0 rx_missed_errors: 0 tx_aborted_errors: 0 tx_carrier_errors: 0 tx_window_errors: 0 tx_abort_late_coll: 0 tx_deferred_ok: 0 tx_single_coll_ok: 0 tx_multi_coll_ok: 0 tx_timeout_count: 0 rx_long_length_errors: 0 rx_short_length_errors: 0 rx_align_errors: 0 tx_tcp_seg_good: 0 tx_tcp_seg_failed: 0 rx_flow_control_xon: 0 rx_flow_control_xoff: 0 tx_flow_control_xon: 0 tx_flow_control_xoff: 0 rx_long_byte_count: 2465341976853 tx_dma_out_of_sync: 0 lro_aggregated: 0 lro_flushed: 0 tx_smbus: 0 rx_smbus: 0 dropped_smbus: 0 os2bmc_rx_by_bmc: 0 os2bmc_tx_by_bmc: 0 os2bmc_tx_by_host: 0 os2bmc_rx_by_host: 0 rx_errors: 0 tx_errors: 0 tx_dropped: 0 rx_length_errors: 0 rx_over_errors: 0 rx_frame_errors: 0 rx_fifo_errors: 0 tx_fifo_errors: 0 tx_heartbeat_errors: 0 tx_queue_0_packets: 3 tx_queue_0_bytes: 210 tx_queue_0_restart: 0 rx_queue_0_packets: 2984012000 rx_queue_0_bytes: 2453405930211 rx_queue_0_drops: 0 rx_queue_0_csum_err: 0 rx_queue_0_alloc_failed: 0 Again, I appreciate any help. Thanks. (Sorry if this is a duplicate).YM_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc _______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
