OK, client 247 is the good one ;-)
It is rather an LNET issue.
All three clients have:
> options lnet networks="o2ib8(ib0),o2ib6(ib0)"
I can mount my nodemapped Lustre, with its MGS on 10.20.6.63@o2ib8, and
another Lustre on o2ib6.
However, the log of the MGS at 10.20.6.63@o2ib8 remarks
> MGS: Connection restored to 15da687f-7a11-4913-9f9d-3c764c97a9cb (at
10.20.3.246@o2ib6)
> MGS: Connection restored to 059da3e0-b277-4767-8678-e83730769fb8 (at
10.20.3.248@o2ib6)
and then
> MGS: Connection restored to a0ee6b8c-48b9-46e8-ba2c-9448889c77ed (at
10.20.3.247@o2ib8)
I can see the "alien" nids also in
mgs # ls /proc/fs/lustre/mgs/MGS/exports
... 10.20.3.246@o2ib6 10.20.3.247@o2ib8 10.20.3.248@o2ib6
Question is: Why would an MGS on "o2ib8" accept connections from a client on
"o2ib6"?
Obviously, this would not happen if the client had only o2ib6. So the MGS is somewhat confused, since
the LNET connection is actually via o2ib8, but the labels and the nodemapping use o2ib6.
This is boot-resistant, the MGS has these wrong nids stored somewhere - can I erase them and start
again with correct nids?
Regards
Thomas
On 6/18/24 17:33, Thomas Roth via lustre-discuss wrote:
Hi all,
what is the meaning of the "exports" property/parameter of a nodemap?
I have
mgs ]# lctl nodemap_info newclients
...
There are three clients:
mgs # lctl get_param nodemap.newclients.ranges
nodemap.newclients.ranges=
[
{ id: 13, start_nid: 10.20.3.246@o2ib8, end_nid: 10.20.3.246@o2ib8 },
{ id: 12, start_nid: 10.20.3.247@o2ib8, end_nid: 10.20.3.247@o2ib8 },
{ id: 9, start_nid: 10.20.3.248@o2ib8, end_nid: 10.20.3.248@o2ib8 }
This nodemap has
nodemap.newclients.admin_nodemap=0
nodemap.newclients.trusted_nodemap=0
nodemap.newclients.deny_unknown=1
and
mgs # lctl get_param nodemap.newclients.exports
nodemap.newclients.exports=
[
{ nid: 10.20.3.247@o2ib8, uuid: 5d9964f9-81eb-4ea5-93dc-a145534f9e74 },
]
_This_ client, 10.20.3.247, behaves differently: No access for root (ok!), no
access for a regular user.
The other two clients, 10.20.3.246/248, show no access for root (ok!), while a regular users sees the
squashed (uid 99) directories of the top level and his own directories/files with correct uid/gid
beneath.
And the only difference between these clients seems to be these 'exports' (totally absent from the
manual, btw).
Regards,
Thomas
--
--------------------------------------------------------------------
Thomas Roth
Department: Informationstechnologie
Location: SB3 2.291
GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de
Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528
Managing Directors / Geschäftsführung:
Professor Dr. Paolo Giubellino, Dr. Katharina Stummeyer, Jörg Blaurock
Chairman of the Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:
Ministerialdirigent Dr. Volkmar Dietz
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
