Hi all,
I was measuring the latency with the new Intel E810. I first used
testpmd application with a single core and a single pair of queues and measured
the latency on the generator side. The problem is that a latency bump occurs
when the background traffic is above a certain threshold. I noticed that the
threshold would move (at different rate of background traffic) depending on the
speed of the recv and xmit function (i.e. bulk, SSE or AVX2)
To identify where the bump occurs, I added hardware timestamp support
to the application. I enabled rx hardware timestamp offload capability of E810,
used rte_eth_timesync_read_time after rte_eth_rx_burst returns and
rte_eth_timesync_read_tx_timestamp after rte_eth_tx_burst returns. I found the
latency bump occurs between the packet arrives at the PHY Core and
rte_eth_tx_burst returns. I also measures the CPU cycles before
rte_eth_rx_burst is called and rte_eth_tx_burst returns in the user space. The
gap in CPU cycles is stable regardless of the background traffic. This means
the bump resides between the packet arrives the NIC and the packet is extracted
from the main memory via rte_eth_rx_burst.
Meanwhile I failed to find any DPDK latency report from Intel nor
mails from those who might experience the same problem. Does anyone meet the
same problem and probably know what happens between the packet is in the PHY
Core and the packet is in the memory? Maybe Intel Validation Team?
I guess it may relate to packet discarding logic in the firmware or the
DMA process. I saw this issue on different servers and different versions of
firmware or DDP as well.
Configuration of the server:
CPU: Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
RAM: DDR4 11 x 32 GB, 2933 MHz, 6 Channels
OS: Ubuntu 20.04.2 LTS
Kernel: 5.4.0-89-generic
Ice kernel driver version: 1.6.7
OS default DDP version: 1.3.26
Firmware version: 3.0
Traffic generator: MoonGen with two Mellanox ConnectX-5 EN 100G NICs
Best Wishes
Pan
