Hi,
The CEA hit this kind of issue but with Lustre router (server 2.12 -> router 2.15). "lctl
ping" from router to the server make the route go "up".
We solve this by setting map_on_demand=1 on the router.
Note here, map_on_demand=1 should be the default on a 2.15 to avoid this kind of
issue: LU-15186 <https://jira.whamcloud.com/browse/LU-15186> Default ko2iblnd
map_on_demand to 1
But, "lnetctl import" stills set map_on_demand=0 by default. This should be solve by
LU-15538 <https://jira.whamcloud.com/browse/LU-15538>/LU-12452
<https://jira.whamcloud.com/browse/LU-12452>.
You can verify this with "lnetctl net show -v" on the server side.
In your case map_on_demand=1 on the client side (2.12) does not work because
the 2.15 server asks for 257 frags but the 2.12 is not able to negotiate with
this value (256 is the maximum for a 2.12 node).
Pinging from the server helps because a 2.15 node is able to negotiate frags (even
with map_on_demand=0, LU-15094 <https://jira.whamcloud.com/browse/LU-15094>):
* 2.15 node initiate the connection with max_frags=257
* 2.12 node rejects the connection and requests max_frags=256
* 2.15 node retries with max_frags=256 and save this value in memory for the
remote peer
* 2.12 node accepts the connection
* 2.12 node initiate the connection with max_frags=256
* 2.15 node accept the connection because it uses the save value of
max_frags=256.
So, setting map_on_demand=1 on the server side (on the 2.15 nodes) should solve
your issue.
Regards,
Etienne
On 7/24/25 22:23, Makia Minich via lustre-discuss wrote:
Recently we upgraded our lustre servers to RHEL 8 with lustre 2.15.5 but due to
scheduling the clients are still currently at RHEL 7 with lustre version
2.12.6. Infiniband is the interconnect.
We've found that the client will fail to mount unless we run a "lnetctl ping" from the server side to the client. Once that happens then the client will ultimately mount. We've seen the following
error in the logs on the server side:
lNet: 1920125:0:(o2iblnd_cd.c:2587:kiblnd_passive_connect()) Can’t accept conn
from 172.16.19.6@o2ib (version 12): max_frags 256 incompatible without FMR_pool
(257 wanted)
Attempting to set map_on_demand on the client side didn't help, resulting in the same error. Are there any parameters or configuration changes that may help the situation? At this time we aren't
able to upgrade the client side to RHEL 8, so we're ultimately limited on available versions, so looking for ideas on what to try next.
Thanks.
_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
