Hi Dariusz, Thanks for the reply. I'm using 68 byte UDP packets with diverse SIP,DIPs, the traffic is balanced between the cores (I included some logging below)
The flows and groups I have are group 0: eth => jump group 2 group 2: eth / ipv4 => rss to sw queues group 2: eth => rss to sw queues (lower priority) group 4: eth => random N => to sw queue 0 group 4: eth => drop (lower priority) Group 4 is not actually referenced from anywhere, I assume this does not affect performance. 2024-10-28 10:25:25 INFO: <0> stats, ipackets:799741213, opackets:799741213 2024-10-28 10:25:25 INFO: <0> flow 0x188195140 grp:pri 0:001 end end => jump jump 2/end end 2024-10-28 10:25:25 INFO: <0> flow 0x188174500 grp:pri 2:548 eth eth/ip4 ip4/end end => count count/rss rss 0-31/end end (799741213, 51183437632) 2024-10-28 10:25:25 INFO: <0> flow 0x1883d73c0 grp:pri 2:549 eth eth/end end => count count/rss rss 0-31/end end (0, 0) 2024-10-28 10:25:25 INFO: <0> flow 0x1883b67c0 grp:pri 4:001 random random/end end => rss rss 0-31/end end 2024-10-28 10:25:25 INFO: <0> flow 0x1883d7140 grp:pri 4:549 end end => drop drop/end end 2024-10-28 10:25:35 INFO: <0> flow 0x188195140 grp:pri 0:001 end end => jump jump 2/end end 2024-10-28 10:25:35 INFO: <0> flow 0x188174500 grp:pri 2:548 eth eth/ip4 ip4/end end => count count/rss rss 0-31/end end (799741213, 51183437632) 2024-10-28 10:25:35 INFO: <0> flow 0x1883d73c0 grp:pri 2:549 eth eth/end end => count count/rss rss 0-31/end end (0, 0) 2024-10-28 10:25:35 INFO: <0> flow 0x1883b67c0 grp:pri 4:001 random random/end end => rss rss 0-31/end end 2024-10-28 10:25:35 INFO: <0> flow 0x1883d7140 grp:pri 4:549 end end => drop drop/end end 2024-10-28 10:25:40 INFO: <0> xstats, rx_good_packets:799741213, tx_good_packets:799741213, rx_good_bytes:51183437632, tx_good_bytes:51183437632, rx_q0_packets:24987083, rx_q0_bytes:1599173312, rx_q1_packets:24988480, rx_q1_bytes:1599262720, rx_q2_packets:24995304, rx_q2_bytes:1599699456, rx_q3_packets:24998711, rx_q3_bytes:1599917504, rx_q4_packets:24989350, rx_q4_bytes:1599318400, rx_q5_packets:24991546, rx_q5_bytes:1599458944, rx_q6_packets:24991647, rx_q6_bytes:1599465408, rx_q7_packets:24995441, rx_q7_bytes:1599708224, rx_q8_packets:24989564, rx_q8_bytes:1599332096, rx_q9_packets:24990980, rx_q9_bytes:1599422720, rx_q10_packets:24996265, rx_q10_bytes:1599760960, rx_q11_packets:24996320, rx_q11_bytes:1599764480, rx_q12_packets:24987707, rx_q12_bytes:1599213248, rx_q13_packets:24983936, rx_q13_bytes:1598971904, rx_q14_packets:24994621, rx_q14_bytes:1599655744, rx_q15_packets:24991660, rx_q15_bytes:1599466240, tx_q0_packets:24987083, tx_q0_bytes:1599173312, tx_q1_packets:24988480, tx_q1_bytes:1599262720, tx_q2_packets:24995304, tx_q2_bytes:1599699456, tx_q3_packets:24998711, tx_q3_bytes:1599917504, tx_q4_packets:24989350, tx_q4_bytes:1599318400, tx_q5_packets:24991546, tx_q5_bytes:1599458944, tx_q6_packets:24991647, tx_q6_bytes:1599465408, tx_q7_packets:24995441, tx_q7_bytes:1599708224, tx_q8_packets:24989564, tx_q8_bytes:1599332096, tx_q9_packets:24990980, tx_q9_bytes:1599422720, tx_q10_packets:24996265, tx_q10_bytes:1599760960, tx_q11_packets:24996320, tx_q11_bytes:1599764480, tx_q12_packets:24987707, tx_q12_bytes:1599213248, tx_q13_packets:24983936, tx_q13_bytes:1598971904, tx_q14_packets:24994621, tx_q14_bytes:1599655744, tx_q15_packets:24991660, tx_q15_bytes:1599466240, rx_unicast_bytes:51183437632, rx_unicast_packets:799741213, tx_unicast_bytes:51183437632, tx_unicast_packets:799741213, tx_phy_packets:799741213, rx_phy_packets:1500000000, rx_prio0_buf_discard_packets:700258787, tx_phy_bytes:54382402484, rx_phy_bytes:102000000000 2024-10-28 10:25:40 INFO: <0> stats, ipackets:799741213, opackets:799741213 ^C2024-10-28 10:25:42 INFO: forwarder13 exiting on core 21 n_rx_pkts: 24987707, n_sample_pkts: 0, max_rx_burst: 36, max_queue_depth: 30 2024-10-28 10:25:42 INFO: forwarder5 exiting on core 13 n_rx_pkts: 24989350, n_sample_pkts: 0, max_rx_burst: 34, max_queue_depth: 30 2024-10-28 10:25:42 INFO: forwarder8 exiting on core 16 n_rx_pkts: 24995441, n_sample_pkts: 0, max_rx_burst: 36, max_queue_depth: 36 2024-10-28 10:25:42 INFO: forwarder14 exiting on core 22 n_rx_pkts: 24983936, n_sample_pkts: 0, max_rx_burst: 36, max_queue_depth: 30 2024-10-28 10:25:42 INFO: forwarder19 exiting on core 27 n_rx_pkts: 24995349, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 34 2024-10-28 10:25:42 INFO: forwarder4 exiting on core 12 n_rx_pkts: 24998711, n_sample_pkts: 0, max_rx_burst: 33, max_queue_depth: 33 2024-10-28 10:25:42 INFO: forwarder7 exiting on core 15 n_rx_pkts: 24991647, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 34 2024-10-28 10:25:42 INFO: forwarder1 exiting on core 9 n_rx_pkts: 24987083, n_sample_pkts: 0, max_rx_burst: 32, max_queue_depth: 30 2024-10-28 10:25:42 INFO: forwarder11 exiting on core 19 n_rx_pkts: 24996265, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 34 2024-10-28 10:25:42 INFO: forwarder15 exiting on core 23 n_rx_pkts: 24994621, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 34 2024-10-28 10:25:42 INFO: forwarder10 exiting on core 18 n_rx_pkts: 24990980, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 32 2024-10-28 10:25:42 INFO: forwarder3 exiting on core 11 n_rx_pkts: 24995304, n_sample_pkts: 0, max_rx_burst: 25, max_queue_depth: 25 2024-10-28 10:25:42 INFO: forwarder2 exiting on core 10 n_rx_pkts: 24988480, n_sample_pkts: 0, max_rx_burst: 32, max_queue_depth: 30 2024-10-28 10:25:42 INFO: forwarder6 exiting on core 14 n_rx_pkts: 24991546, n_sample_pkts: 0, max_rx_burst: 34, max_queue_depth: 30 2024-10-28 10:25:42 INFO: forwarder12 exiting on core 20 n_rx_pkts: 24996320, n_sample_pkts: 0, max_rx_burst: 28, max_queue_depth: 28 2024-10-28 10:25:42 INFO: forwarder9 exiting on core 17 n_rx_pkts: 24989564, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 34 2024-10-28 10:25:42 INFO: forwarder17 exiting on core 25 n_rx_pkts: 24986227, n_sample_pkts: 0, max_rx_burst: 36, max_queue_depth: 32 2024-10-28 10:25:42 INFO: forwarder20 exiting on core 28 n_rx_pkts: 24994610, n_sample_pkts: 0, max_rx_burst: 37, max_queue_depth: 35 2024-10-28 10:25:42 INFO: forwarder16 exiting on core 24 n_rx_pkts: 24991660, n_sample_pkts: 0, max_rx_burst: 34, max_queue_depth: 34 2024-10-28 10:25:42 INFO: forwarder26 exiting on core 34 n_rx_pkts: 24994289, n_sample_pkts: 0, max_rx_burst: 29, max_queue_depth: 24 2024-10-28 10:25:42 INFO: forwarder23 exiting on core 31 n_rx_pkts: 24986582, n_sample_pkts: 0, max_rx_burst: 34, max_queue_depth: 34 2024-10-28 10:25:42 INFO: forwarder22 exiting on core 30 n_rx_pkts: 24995067, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 34 2024-10-28 10:25:42 INFO: forwarder21 exiting on core 29 n_rx_pkts: 24994793, n_sample_pkts: 0, max_rx_burst: 29, max_queue_depth: 23 2024-10-28 10:25:42 INFO: forwarder31 exiting on core 39 n_rx_pkts: 24993460, n_sample_pkts: 0, max_rx_burst: 26, max_queue_depth: 22 2024-10-28 10:25:42 INFO: forwarder29 exiting on core 37 n_rx_pkts: 24987219, n_sample_pkts: 0, max_rx_burst: 30, max_queue_depth: 30 2024-10-28 10:25:42 INFO: forwarder24 exiting on core 32 n_rx_pkts: 24986863, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 34 2024-10-28 10:25:42 INFO: forwarder32 exiting on core 40 n_rx_pkts: 24991661, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 34 2024-10-28 10:25:42 INFO: forwarder18 exiting on core 26 n_rx_pkts: 24991276, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 36 2024-10-28 10:25:42 INFO: forwarder30 exiting on core 38 n_rx_pkts: 24989407, n_sample_pkts: 0, max_rx_burst: 30, max_queue_depth: 30 2024-10-28 10:25:42 INFO: forwarder25 exiting on core 33 n_rx_pkts: 24994537, n_sample_pkts: 0, max_rx_burst: 26, max_queue_depth: 22 2024-10-28 10:25:42 INFO: forwarder28 exiting on core 36 n_rx_pkts: 24996627, n_sample_pkts: 0, max_rx_burst: 26, max_queue_depth: 22 2024-10-28 10:25:42 INFO: forwarder27 exiting on core 35 n_rx_pkts: 24994631, n_sample_pkts: 0, max_rx_burst: 34, max_queue_depth: 32 On Fri, Oct 25, 2024 at 8:03 AM Dariusz Sosnowski <dsosnow...@nvidia.com> wrote: > Hi Tony, > > I apologize for the late response. > > > From: Tony Hart <tony.h...@domainhart.com> > > Sent: Saturday, October 12, 2024 17:09 > > To: users@dpdk.org > > Subject: mlx5: imissed versus prio0_buf_discards > > > > External email: Use caution opening links or attachments > > > > > > > > I have a simple DPDK app that receives packets via RSS from a CX7 > (400G). The app uses 16 queues across 16 cores. What I see is dropped > packets even at only 50Mpps. > > > > Looking at rte_eth_port_xstats() I see rx_prio0_buf_discard_packets > matches the number of packets dropped however the imissed counter (from > rte_eth_port_stats) is 0. Indeed when I look at the rx_queue depths from > each thread in the app they barely reach 30 entries (I'm using the default > number of queue descs). > > > > What is the difference between rx_prio0_buf_discards and imissed > counters, why would rx_prio0_buf_discards increase but not imissed? > > Both counters measure packet drops, but at different levels: > > - imissed - Measures drops caused by lack of free descriptors in the Rx > queue. > This indicates that SW cannot keep up with current packet rate. > - rx_prio0_buf_discards - Measures drops caused by lack of free space in > NIC's Rx buffer. > This indicates that HW cannot keep up with current packet rate. > > What kind of traffic are you generating? > What kind of flow tables and rules do you create? > In your application, do you see that packets are roughly equally > distributed across all 16 Rx queues? > > > > > many thanks, > > tony > > > > fyi: this is using DPKD 24.07 and the HWS RTE FLOW Api to setup the RSS > flow. Firmware is 28.41 > > Best regards, > Dariusz Sosnowski > -- tony
