Hi Beilei, On Wed, Jul 20, 2016 at 3:59 AM, Xing, Beilei <beilei.xing at intel.com> wrote: > Hi Ceara, > >> -----Original Message----- >> From: Take Ceara [mailto:dumitru.ceara at gmail.com] >> Sent: Tuesday, July 19, 2016 10:59 PM >> To: Xing, Beilei <beilei.xing at intel.com> >> Cc: Zhang, Helin <helin.zhang at intel.com>; Wu, Jingjing >> <jingjing.wu at intel.com>; dev at dpdk.org >> Subject: Re: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 >> NICs for some RX mbuf sizes >> >> Hi Beilei, >> >> I changed the way I run testmpd to: >> >> testpmd -c 0x331 -w 0000:82:00.0 -w 0000:83:00.0 -- --mbuf-size 1152 >> --rss-ip - >> -rxq=2 --txpkts 1024 -i >> >> As far as I understand this will allocate mbufs with the same size I was >> using >> in my test (--mbuf-size seems to include the mbuf headroom therefore 1152 >> = 1024 + 128 headroom). >> >> testpmd> start tx_first >> io packet forwarding - CRC stripping disabled - packets/burst=32 >> nb forwarding cores=1 - nb forwarding ports=2 >> RX queues=2 - RX desc=128 - RX free threshold=32 >> RX threshold registers: pthresh=8 hthresh=8 wthresh=0 >> TX queues=1 - TX desc=512 - TX free threshold=32 >> TX threshold registers: pthresh=32 hthresh=0 wthresh=0 >> TX RS bit threshold=32 - TXQ flags=0xf01 >> testpmd> show port stats all >> >> ######################## NIC statistics for port 0 >> ######################## >> RX-packets: 18817613 RX-missed: 5 RX-bytes: 19269115888 >> RX-errors: 0 >> RX-nombuf: 0 >> TX-packets: 18818064 TX-errors: 0 TX-bytes: 19269567464 >> >> ########################################################## >> ################## >> >> ######################## NIC statistics for port 1 >> ######################## >> RX-packets: 18818392 RX-missed: 5 RX-bytes: 19269903360 >> RX-errors: 0 >> RX-nombuf: 0 >> TX-packets: 18817979 TX-errors: 0 TX-bytes: 19269479424 >> >> ########################################################## >> ################## >> >> Ttraffic is sent/received. However, I couldn't find any way to verify that >> the >> incoming mbufs actually have the mbuf->hash.rss field set except for starting >> test-pmd with gdb and setting a breakpoint in the io fwd engine. After doing >> that I noticed that none of the incoming packets has the PKT_RX_RSS_HASH >> flag set in ol_flags... I guess for some reason test-pmd doesn't actually >> configure RSS in this case but I fail to see where. >> > > Actually there's a way to check mbuf->hash.rss, you need set forward mode to > "rxonly", and set verbose to 1. > I run testpmd with the configuration you used, and found i40e RSS works well. > With the following steps, you can see RSS hash value and receive queue, and > PKT_RX_RSS_HASH is set too. > I think you can use the same way to check what you want. > > ./testpmd -c fffff -n 4 -- -i --coremask=0xffffe --rxq=16 --txq=16 > --mbuf-size 1152 --rss-ip --txpkts 1024 > testpmd> set verbose 1 > testpmd> set fwd rxonly > testpmd> start > testpmd> port 0/queue 1: received 1 packets > src=00:00:01:00:0F:00 - dst=68:05:CA:32:03:4C - type=0x0800 - length=1020 - > nb > - Receive queue=0x1 > PKT_RX_RSS_HASH > port 0/queue 0: received 1 packets > src=00:00:01:00:0F:00 - dst=68:05:CA:32:03:4C - type=0x0800 - length=1020 - > nb_segs=1 - RSS hash=0x4e949f23 - RSS queue=0x0Unknown packet type > - Receive queue=0x0 > PKT_RX_RSS_HASH > port 0/queue 8: received 1 packets > src=00:00:01:00:0F:00 - dst=68:05:CA:32:03:4C - type=0x0800 - length=1020 - > nb_segs=1 - RSS hash=0xa3c78b2b - RSS queue=0x8Unknown packet type > - Receive queue=0x8 > PKT_RX_RSS_HASH > port 0/queue 5: received 1 packets > src=00:00:01:00:0F:00 - dst=68:05:CA:32:03:4C - type=0x0800 - length=1020 - > nb_segs=1 - RSS hash=0xe29b3d36 - RSS queue=0x5Unknown packet type > - Receive queue=0x5 > PKT_RX_RSS_HASH >
Following your testpmd example run I managed to replicate the problem on my dpdk 16.04 setup like this: I have two X710 adapters connected back to back: $ ./tools/dpdk_nic_bind.py -s Network devices using DPDK-compatible driver ============================================ 0000:01:00.3 'Ethernet Controller X710 for 10GbE SFP+' drv=igb_uio unused= 0000:81:00.3 'Ethernet Controller X710 for 10GbE SFP+' drv=igb_uio unused= The firmware of the two adapters is up to date with the latest version: 5.04 (f5.0.40043 a1.5 n5.04 e24cd) I run testpmd with mbuf-size 1152 and txpktsize 1024 such that upon receival the whole mbuf (except headroom) is filled. I enabled RX IP checksum in hw and RX RSS hashing for UDP. With test-pmd forward mode "rxonly" and verbose 1 I see that incoming packets have PKT_RX_RSS_HASH set but the hash value is 0. ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i --coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 1152 --txpkts 1024 --enable-rx-cksum --rss-udp [...] testpmd> set verbose 1 Change verbose level from 0 to 1 testpmd> set fwd rxonly Set rxonly packet forwarding mode testpmd> start tx_first rxonly packet forwarding - CRC stripping disabled - packets/burst=32 nb forwarding cores=16 - nb forwarding ports=2 RX queues=16 - RX desc=128 - RX free threshold=32 RX threshold registers: pthresh=8 hthresh=8 wthresh=0 TX queues=16 - TX desc=512 - TX free threshold=32 TX threshold registers: pthresh=32 hthresh=0 wthresh=0 TX RS bit threshold=32 - TXQ flags=0xf01 port 0/queue 1: received 32 packets src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - length=1024 - nb_segs=1 - RSS hash=0x0 - RSS queue=0x1 - (outer) L2 type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN - (outer) L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown - Inner L3 type: Unknown - Inner L4 type: Unknown - Receive queue=0x1 PKT_RX_RSS_HASH src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - length=1024 - nb_segs=1 - RSS hash=0x0 - RSS queue=0x1 - (outer) L2 type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN - (outer) L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown - Inner L3 type: Unknown - Inner L4 type: Unknown - Receive queue=0x1 PKT_RX_RSS_HASH If I use a different mbuf-size, for example 2048, everything is fine: ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i --coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 2048 --txpkts 1024 --enable-rx-cksum --rss-udp [...] testpmd> set verbose 1 Change verbose level from 0 to 1 testpmd> set fwd rxonly Set rxonly packet forwarding mode testpmd> start tx_first rxonly packet forwarding - CRC stripping disabled - packets/burst=32 nb forwarding cores=16 - nb forwarding ports=2 RX queues=16 - RX desc=128 - RX free threshold=32 RX threshold registers: pthresh=8 hthresh=8 wthresh=0 TX queues=16 - TX desc=512 - TX free threshold=32 TX threshold registers: pthresh=32 hthresh=0 wthresh=0 TX RS bit threshold=32 - TXQ flags=0xf01 port 0/queue 1: received 32 packets src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - length=1024 - nb_segs=1 - RSS hash=0x5263c3f2 - RSS queue=0x1 - (outer) L2 type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN - (outer) L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown - Inner L3 type: Unknown - Inner L4 type: Unknown - Receive queue=0x1 PKT_RX_RSS_HASH Another weird thing I noticed is when I run test-pmd without --enable-rx-cksum (which is the default mode) then the RSS flag doesn get set at all. Instead the PKT_RX_FDIR flag gets set. This happens even though fdir_mode is set to RTE_FDIR_MODE_NONE in the device configuration: ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i --coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 1152 --txpkts 1024 --rss-udp [...] testpmd> set verbose 1 Change verbose level from 0 to 1 testpmd> set fwd rxonly Set rxonly packet forwarding mode testpmd> start tx_first rxonly packet forwarding - CRC stripping disabled - packets/burst=32 nb forwarding cores=16 - nb forwarding ports=2 RX queues=16 - RX desc=128 - RX free threshold=32 RX threshold registers: pthresh=8 hthresh=8 wthresh=0 TX queues=16 - TX desc=512 - TX free threshold=32 TX threshold registers: pthresh=32 hthresh=0 wthresh=0 TX RS bit threshold=32 - TXQ flags=0xf01 port 0/queue 1: received 16 packets src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - length=1024 - nb_segs=2 - FDIR matched hash=0xc3f2 ID=0x5263 Unknown packet type - Receive queue=0x1 PKT_RX_FDIR src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - length=1024 - nb_segs=2 - FDIR matched hash=0xc3f2 ID=0x5263 Unknown packet type - Receive queue=0x1 PKT_RX_FDIR Please let me know if there's more debug info that might be of interest in troubleshooting the RSS=0 issue. Thanks, Dumitru > /Beilei > >> Thanks, >> Dumitru >>