Hi Adrian,
As promised, I researched the Connect4-LX NIC issues with Debian that we
experienced during our initial S7-2 kernel testing. Recommend you pick
up this patch, fixes numerous Mellanox ConnectX4-LX related Tainted
Kernel messages and extremely slow performance on the S2-2. Modified
my 6.17.2 Linux kernel on the S7-2 to test this patch, and it works.
Also, I believe this patch may resolve this issue for other
architectures/systems as well. No more tainted kernel messages.
The problem: Incorrect comparision calculation with "WARN_ON(PAGE_SIZE
!= 2 * DEFAULT_FRAG_SIZE); "
tonyr@s7-debian:~$ getconf PAGE_SIZE
8192
Example:
DEFAULT_FRAG_SIZE is 2048
"WARN_ON(8192 != 2 * 2048); " == 4096 doesn't match 8192 and creates an
issue.
To resolve:
-#define DEFAULT_FRAG_SIZE (2048)
+#define DEFAULT_FRAG_SIZE (PAGE_SIZE / 2)
Tests Performed:
iperf3
ssh and sftp large file transfers
No Tainted kernel messages and fast file transfers
PS. - The Connextx4-LX 10/25GB NIC also works with both Solaris 11.4 and
Debian for sparc64, so it definitely recommended.
Test-By: Tony Rodriguez [email protected]
Regards,
Tony Rodriguez
On 10/14/25 10:18 AM, Tony Rodriguez wrote:
https://lkml.org/lkml/2025/9/2/1794
Date Tue, 2 Sep 2025 11:56:06 -0700
From Saeed Mahameed <>
Subject Re: [PATCH] net/mlx5e: Make DEFAULT_FRAG_SIZE relative to
page size
On 02 Sep 21:00, Mingrui Cui wrote:
>When page size is 4K, DEFAULT_FRAG_SIZE of 2048 ensures that with 3
>fragments per WQE, odd-indexed WQEs always share the same page with
>their subsequent WQE. However, this relationship does not hold for page
>sizes larger than 8K. In this case, wqe_index_mask cannot guarantee that
>newly allocated WQEs won't share the same page with old WQEs.
>
>If the last WQE in a bulk processed by mlx5e_post_rx_wqes() shares a
>page with its subsequent WQE, allocating a page for that WQE will
>overwrite mlx5e_frag_page, preventing the original page from being
>recycled. When the next WQE is processed, the newly allocated page will
>be immediately recycled.
>
>In the next round, if these two WQEs are handled in the same bulk,
>page_pool_defrag_page() will be called again on the page, causing
>pp_frag_count to become negative.
>
>Fix this by making DEFAULT_FRAG_SIZE always equal to half of the page
>size.
>
>Signed-off-by: Mingrui Cui <[email protected]>
CC: Dragos Tatulea <[email protected]>
Dragos is on a mission to improve page_size support in mlx5.
Dragos, please look into this, I am not sure making DEFAULT_FRAG_SIZE
dependant on PAGE_SIZE is the correct way to go,
see mlx5e_build_rq_frags_info()
I believe we don't do page flipping for > 4k pages, but I might be wrong,
anyway the code also should throw a warn_on:
/* The last fragment of WQE with index 2*N may share the page with the
* first fragment of WQE with index 2*N+1 in certain cases. If WQE
* 2*N+1
* is not completed yet, WQE 2*N must not be allocated, as it's
* responsible for allocating a new page.
*/
if (frag_size_max == PAGE_SIZE) {
/* No WQE can start in the middle of a page. */
info->wqe_index_mask = 0;
} else {
/* PAGE_SIZEs starting from 8192 don't use 2K-sized fragments,
* because there would be more than MLX5E_MAX_RX_FRAGS of
* them.
*/
WARN_ON(PAGE_SIZE != 2 * DEFAULT_FRAG_SIZE);
/* Odd number of fragments allows to pack the last fragment of
* the previous WQE and the first fragment of the next WQE
* into
* the same page.
* As long as DEFAULT_FRAG_SIZE is 2048, and
* MLX5E_MAX_RX_FRAGS
* is 4, the last fragment can be bigger than the rest only
* if
* it's the fourth one, so WQEs consisting of 3 fragments
* will
* always share a page.
* When a page is shared, WQE bulk size is 2, otherwise
* just 1.
*/
info->wqe_index_mask = info->num_frags % 2;
}
Looking at the above makes me think that this patch is correct, but a
more
careful look is needed to be taken, a Fixes tag is also required and
target
'net' branch.
Thanks,
Saeed.
>---
> drivers/net/ethernet/mellanox/mlx5/core/en/params.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
>diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
>index 3cca06a74cf9..d96a3cbea23c 100644
>--- a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
>+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
>@@ -666,7 +666,7 @@ static void
mlx5e_rx_compute_wqe_bulk_params(struct mlx5e_params *params,
> info->refill_unit = DIV_ROUND_UP(info->wqe_bulk, split_factor);
> }
>
>-#define DEFAULT_FRAG_SIZE (2048)
>+#define DEFAULT_FRAG_SIZE (PAGE_SIZE / 2)
>
> static int mlx5e_build_rq_frags_info(struct mlx5_core_dev *mdev,
> struct mlx5e_params *params,
>--
>2.43.0
>
>
