Hello,
As advised, I tested hardware timestamps with LRO enabled on our ConnectX-7
NICs. However, the timestamps of LROed packets still show inconsistent and
abnormally large gaps from normal packets.
Interestingly, I found this issue does not appear on all CX-7 NICs. Even with
identical DPDK code, firmware version (28.43.2566), and hardware models from
the same manufacturer, only specific NICs exhibit this inconsistency.
I have confirmed that:
• All NICs use the same driver and firmware version.
• All NICs are of the same model (MCX75310AAS-NEA_Ax).
• The issue occurs only when LRO is enabled together with RX hardware
timestamping.
• Disabling LRO eliminates the issue.
I would appreciate any insight into how this behavior can occur on only some
ports despite same software and hardware setup.
Below is my code snippet.
```c
/*----------------------------------------------------------------------------*/
static inline int
is_timestamp_enabled(struct rte_mbuf *mbuf)
{
static uint64_t timestamp_rx_dynflag = 0;
int timestamp_rx_dynflag_offset;
if (!timestamp_rx_dynflag)
{
timestamp_rx_dynflag_offset =
rte_mbuf_dynflag_lookup(RTE_MBUF_DYNFLAG_RX_TIMESTAMP_NAME, NULL);
if (timestamp_rx_dynflag_offset < 0)
{
return 0;
}
timestamp_rx_dynflag = RTE_BIT64(timestamp_rx_dynflag_offset);
}
return mbuf->ol_flags & timestamp_rx_dynflag;
}
/*----------------------------------------------------------------------------*/
static inline rte_mbuf_timestamp_t *
get_timestamp(struct rte_mbuf *mbuf)
{
static int timestamp_dynfield_offset = -1;
if (timestamp_dynfield_offset < 0)
{
timestamp_dynfield_offset =
rte_mbuf_dynfield_lookup(RTE_MBUF_DYNFIELD_TIMESTAMP_NAME, NULL);
if (timestamp_dynfield_offset < 0)
{
return 0;
}
}
return RTE_MBUF_DYNFIELD(mbuf,
timestamp_dynfield_offset,
rte_mbuf_timestamp_t *);
}
/*----------------------------------------------------------------------------*/
static inline rte_mbuf_timestamp_t *
get_rx_hw_timestamp(struct rte_mbuf *pkt)
{
if (!is_timestamp_enabled(pkt))
{
printf("rx_hw_timestamp not enabled in mbuf!\n");
return NULL;
}
return get_timestamp(pkt);
}
```
My DPDK application prints logs as below.
```c
/* parse HW timestamp */
rte_mbuf_timestamp_t *rx_timestamp = get_rx_hw_timestamp(pkt);
printf("[port %d] RX HW timestamp: %#016lx %s\n",
pctx->port_id,
*rx_timestamp,
pkt->ol_flags & PKT_RX_LRO ? "(LROed)" : "(not LROed)");
```
Below are observations from two CX-7 ports under identical conditions.
Normal NIC (port 0):
[port 0] RX HW timestamp: 0x00007dcd2d185b (LROed)
[port 0] RX HW timestamp: 0x00007dcd2d1911 (LROed)
[port 0] RX HW timestamp: 0x00007dcd2d19c9 (LROed)
[port 0] RX HW timestamp: 0x00007dcd2d37ca (LROed)
[port 0] RX HW timestamp: 0x00007dcd2d4cb3 (not LROed)
[port 0] RX HW timestamp: 0x00007dcd2d4cb3 (not LROed)
[port 0] RX HW timestamp: 0x00007dcd30e019 (not LROed)
[port 0] RX HW timestamp: 0x00007dcd3280bb (not LROed)
Erroneous NIC (port 1):
Below is erroneous NIC's timestamp.
[port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
[port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
[port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
[port 1] RX HW timestamp: 0x3e6eef91bc19f0fd (LROed)
[port 1] RX HW timestamp: 0x000080691b7557 (not LROed)
[port 1] RX HW timestamp: 0x000080691e2311 (not LROed)
[port 1] RX HW timestamp: 0x00008069357553 (not LROed)
[port 1] RX HW timestamp: 0x0000806936e8c1 (not LROed)
As shown above, non-LRO packets consistently have normal hardware timestamps on
both NICs. However, on port 1, all LROed packets return a fixed, invalid
timestamp (0x3e6eef91bc19f0fd), which is clearly inconsistent.
I have also confirmed that other dynfields (rather than dynfield[1] and
dynfield[2]) are unused.
Sincerely,
Junghan Yoon
On Jul 22, 2025, 5:31 PM +0900, Ivan Malov <ivan.ma...@arknetworks.am>, wrote:
> Hello,
>
> On Tue, 22 Jul 2025, Yoon Junghan wrote:
>
> > Hello,
> >
> > I'm currently using DPDK 20.11 with a ConnectX-7 NIC, and I'm trying to
> > retrieve RX hardware timestamps using
> > `rte_mbuf_dyn_rx_timestamp_register()`.
>
> Does the application invoke 'rte_mbuf_dyn_rx_timestamp_register' on its own?
> If
> yes, consider to replace this with invocations of APIs [1] (with field name
> [2])
> and [3] (with flag name [4]). For an example, please refer to [5] and [6].
>
> This is because, as per [7], the driver in question might 'register' the field
> and the flag on its own, in response to 'DEV_RX_OFFLOAD_TIMESTAMP' request,
> so,
> the user application should look up the field/flag, not 'register' it afresh.
>
> If this does not help, then consider to clarify whether the timestamps are
> accurate (and whether the flag is seen in the mbufs) when LRO is not enabled.
>
> [1]
> https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a6adf9b352a83e7d521fd6aa04e305b1c
> [2]
> https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a5159b2d34fa801d171ed0ccce451121b
> [3]
> https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a89d835027034f76a27eb2afe7987ae35
> [4]
> https://doc.dpdk.org/api-20.11/rte__mbuf__dyn_8h.html#a831d7066c7193788351797a65186848a
> [5]
> https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/app/test-pmd/util.c#L44
> [6]
> https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/app/test-pmd/util.c#L60
> [7]
> https://github.com/DPDK/dpdk/blob/d69724b1dcc69784bcef00b96597469b7f6e6207/drivers/net/mlx5/mlx5_rxq.c#L1743
>
> Thank you.
>
> >
> > When LRO is enabled, I notice that LROed mbufs seem to share identical
> > timestamp values, and the timestamps are unexpectedly large or
> > inconsistent. This raises the question of whether
> > LRO is interfering with the correctness of the RX HW timestamps.
> >
> > I’d appreciate any clarification on whether HW RX timestamping is reliable
> > when LRO is enabled on this platform, or if LRO should be just disabled for
> > accurate per-packet timestamping.
> >
> >
> > Sincerely,
> > Junghan Yoon
> >
