On Wed, 23 Jul 2025, Yoon Junghan wrote:
Thank you for quick response.
1) They are different NICs. Not in the same board. Separate adapters in
different PCIe slots.
2) My DPDK app uses 4 separate ports; port 0, port 1, port 2, and port 3. They
are all on different boards. Thus, they are running at the same time.
Excellent. I apologise for one more dumb question, but does isolating the very
specific NIC (so that DPDK does not grab the other ones) that is known to give
strange timestamps, result in the same/unexpected behaviour? Just to make sure.
Thank you.
Sincerely,
Junghan Yoon
On 2025년 7월 23일 PM 10:09 +0900, Ivan Malov <ivan.ma...@arknetworks.am>, wrote:
Hello,
On Wed, 23 Jul 2025, Yoon Junghan wrote:
Hello,
As advised, I tested hardware timestamps with LRO enabled on our
ConnectX-7 NICs. However, the timestamps of LROed packets still show
inconsistent and abnormally large
gaps from normal
packets.
Interestingly, I found this issue does not appear on all CX-7 NICs. Even
with identical DPDK code, firmware version (28.43.2566), and hardware models
from the same
manufacturer, only
specific NICs exhibit this inconsistency.
I have confirmed that:
* All NICs use the same driver and firmware version.
* All NICs are of the same model (MCX75310AAS-NEA_Ax).
1) Do the two "NICs" ('port 0' and 'port 1' from below printout) represent two
different ports/PFs of the same physical 'board'/'adapter card' in fact?
2) If (1) is true, were the results obtained by running the application on both
ports simultaneously (both managed by the DPDK at the same time)?
(just to clarify, -- I'm confused by the fact that the NIC driver itself seems
to invoke 'rte_mbuf_dyn_rx_timestamp_register' for each new RxQ rather than call
it once and then look-up and reuse the existing offsets for more ports/queue ).
Thank you.
* The issue occurs only when LRO is enabled together with RX hardware
timestamping.
* Disabling LRO eliminates the issue.
I would appreciate any insight into how this behavior can occur on only
some ports despite same software and hardware setup.
Below is my code snippet.
```c
/*----------------------------------------------------------------------------*/
static inline int
is_timestamp_enabled(struct rte_mbuf *mbuf)
{
static uint64_t timestamp_rx_dynflag = 0;
int timestamp_rx_dynflag_offset;
if (!timestamp_rx_dynflag)
{
timestamp_rx_dynflag_offset =
rte_mbuf_dynflag_lookup(RTE_MBUF_DYNFLAG_RX_TIMESTAMP_NAME,
NULL);
if (timestamp_rx_dynflag_offset < 0)
{
return 0;
}
timestamp_rx_dynflag = RTE_BIT64(timestamp_rx_dynflag_offset);
}
return mbuf->ol_flags & timestamp_rx_dynflag;
}
/*----------------------------------------------------------------------------*/
static inline rte_mbuf_timestamp_t *
get_timestamp(struct rte_mbuf *mbuf)
{
static int timestamp_dynfield_offset = -1;
if (timestamp_dynfield_offset < 0)
{
timestamp_dynfield_offset =
rte_mbuf_dynfield_lookup(RTE_MBUF_DYNFIELD_TIMESTAMP_NAME,
NULL);
if (timestamp_dynfield_offset < 0)
{
return 0;
}
}
return RTE_MBUF_DYNFIELD(mbuf,
timestamp_dynfield_offset,
rte_mbuf_timestamp_t *);
}
/*----------------------------------------------------------------------------*/
static inline rte_mbuf_timestamp_t *
get_rx_hw_timestamp(struct rte_mbuf *pkt)
{
if (!is_timestamp_enabled(pkt))
{
printf("rx_hw_timestamp not enabled in mbuf!\n");
return NULL;
}
return get_timestamp(pkt);
}
```
My DPDK application prints logs as below.
```c
/* parse HW timestamp */
rte_mbuf_timestamp_t *rx_timestamp = get_rx_hw_timestamp(pkt);
printf("[port %d] RX HW timestamp: %#016lx %s\n",
pctx->port_id,
*rx_timestamp,
pkt->ol_flags & PKT_RX_LRO ? "(LROed)" : "(not LROed)");
```
Below are observations from two CX-7 ports under identical conditions.
Normal NIC (port 0):
[port 0] RX HW timestamp: 0x00007dcd2d185b (LROed)
[port 0] RX HW timestamp: 0x00007dcd2d1911 (LROed)
[port 0] RX HW timestamp: 0x00007dcd2d19c9 (LROed)
[port 0] RX HW timestamp: 0x00007dcd2d37ca (LROed)
[port 0] RX HW timestamp: 0x00007dcd2d4cb3 (not LROed)
[port 0] RX HW timestamp: 0x00007dcd2d4cb3 (not LROed)
[port 0] RX HW timestamp: 0x00007dcd30e019 (not LROed)
[port 0] RX HW timestamp: 0x00007dcd3280bb (not LROed)
Erroneous NIC (port 1):
Below is erroneous NIC's timestamp.
[port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
[port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
[port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
[port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
[port 1] RX HW timestamp: 0x000080691b7557 (not LROed)
[port 1] RX HW timestamp: 0x000080691e2311 (not LROed)
[port 1] RX HW timestamp: 0x00008069357553 (not LROed)
[port 1] RX HW timestamp: 0x0000806936e8c1 (not LROed)
As shown above, non-LRO packets consistently have normal hardware
timestamps on both NICs. However, on port 1, all LROed packets return a fixed,
invalid timestamp
(0x3e6eef91bc19f0fd),
which is clearly inconsistent.
I have also confirmed that other dynfields (rather than dynfield[1] and
dynfield[2]) are unused.
Sincerely,
Junghan Yoon
On Jul 22, 2025, 5:31 PM +0900, Ivan Malov <ivan.ma...@arknetworks.am>,
wrote:
Hello,
On Tue, 22 Jul 2025, Yoon Junghan wrote:
Hello,
I'm currently using DPDK 20.11 with a ConnectX-7 NIC, and I'm trying to
retrieve RX hardware timestamps using `rte_mbuf_dyn_rx_timestamp_register()`.
Does the application invoke 'rte_mbuf_dyn_rx_timestamp_register' on its
own? If
yes, consider to replace this with invocations of APIs [1] (with field
name [2])
and [3] (with flag name [4]). For an example, please refer to [5] and [6].
This is because, as per [7], the driver in question might 'register' the
field
and the flag on its own, in response to 'DEV_RX_OFFLOAD_TIMESTAMP'
request, so,
the user application should look up the field/flag, not 'register' it
afresh.
If this does not help, then consider to clarify whether the timestamps are
accurate (and whether the flag is seen in the mbufs) when LRO is not
enabled.
[1]
https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a6adf9b352a83e7d521fd6aa04e305b1c
[2]
https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a5159b2d34fa801d171ed0ccce451121b
[3]
https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a89d835027034f76a27eb2afe7987ae35
[4]
https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a831d7066c7193788351797a65186848a
[5]
https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/app/test-pmd/util.c#L44
[6]
https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/app/test-pmd/util.c#L60
[7]
https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/drivers/net/mlx5/mlx5_rxq.c#L1743
Thank you.
When LRO is enabled, I notice that LROed mbufs seem to share identical
timestamp values, and the timestamps are unexpectedly large or inconsistent.
This raises
the question of whether
LRO is interfering with the correctness of the RX HW timestamps.
I’d appreciate any clarification on whether HW RX timestamping is
reliable when LRO is enabled on this platform, or if LRO should be just
disabled for accurate
per-packet timestamping.
Sincerely,
Junghan Yoon
