from:"Thomas Roth via lustre\-discuss"

Re: [lustre-discuss] nodemap exports

2024-06-19 Thread Thomas Roth via lustre-discuss


Thanks Sebastien,
this put all clients into the correct LNet. And nodemap.exports looks as 
expected.

(Had to add 'options lnet lnet_peer_discovery_disabled=1' for ‘-o 
network=’ to work).


Regards
Thomas


On 6/19/24 08:31, Sebastien Buisson wrote:

Hi,

I am not sure the NIDs displayed under ‘exports’ and in the logs are the actual 
NIDs used to communicate with, maybe they are just identifiers to describe the 
peers.

If you want to restrict a client to a given LNet network for a given mount point, you 
can use the ‘-o network=’ mount option, like ‘-o network=o2ib8’ in your 
case.

Cheers,
Sebastien.


Le 18 juin 2024 à 18:01, Thomas Roth via lustre-discuss 
 a écrit :

OK, client 247 is the good one ;-)

It is rather an LNET issue.

All three clients have:


options lnet networks="o2ib8(ib0),o2ib6(ib0)"


I can mount my nodemapped Lustre, with its  MGS on 10.20.6.63@o2ib8, and 
another Lustre on o2ib6.

However, the log of the MGS at 10.20.6.63@o2ib8 remarks

MGS: Connection restored to 15da687f-7a11-4913-9f9d-3c764c97a9cb (at 
10.20.3.246@o2ib6)
MGS: Connection restored to 059da3e0-b277-4767-8678-e83730769fb8 (at 
10.20.3.248@o2ib6)

and then

MGS: Connection restored to a0ee6b8c-48b9-46e8-ba2c-9448889c77ed (at 
10.20.3.247@o2ib8)



I can see the "alien" nids also in
mgs # ls /proc/fs/lustre/mgs/MGS/exports
... 10.20.3.246@o2ib6  10.20.3.247@o2ib8  10.20.3.248@o2ib6

Question is: Why would an MGS on "o2ib8" accept connections from a client on 
"o2ib6"?

Obviously, this would not happen if the client had only o2ib6. So the MGS is 
somewhat confused, since the LNET connection is actually via o2ib8, but the 
labels and the nodemapping use o2ib6.



This is boot-resistant, the MGS has these wrong nids stored somewhere - can I 
erase them and start again with correct nids?


Regards
Thomas



On 6/18/24 17:33, Thomas Roth via lustre-discuss wrote:

Hi all,
what is the meaning of the "exports" property/parameter of a nodemap?
I have
mgs ]# lctl nodemap_info newclients
...
There are three clients:
mgs # lctl get_param nodemap.newclients.ranges
nodemap.newclients.ranges=
[
  { id: 13, start_nid: 10.20.3.246@o2ib8, end_nid: 10.20.3.246@o2ib8 },
  { id: 12, start_nid: 10.20.3.247@o2ib8, end_nid: 10.20.3.247@o2ib8 },
  { id: 9, start_nid: 10.20.3.248@o2ib8, end_nid: 10.20.3.248@o2ib8 }
This nodemap has
nodemap.newclients.admin_nodemap=0
nodemap.newclients.trusted_nodemap=0
nodemap.newclients.deny_unknown=1
and
mgs # lctl get_param nodemap.newclients.exports
nodemap.newclients.exports=
[
  { nid: 10.20.3.247@o2ib8, uuid: 5d9964f9-81eb-4ea5-93dc-a145534f9e74 },
]
_This_ client, 10.20.3.247, behaves differently: No access for root (ok!), no 
access for a regular user.
The other two clients, 10.20.3.246/248, show no access for root (ok!), while a 
regular users sees the squashed (uid 99) directories of the top level and his 
own directories/files with correct uid/gid beneath.
And the only difference between these clients seems to be these 'exports' 
(totally absent from the manual, btw).
Regards,
Thomas


--

Thomas Roth
Department: Informationstechnologie
Location: SB3 2.291


GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de

Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528
Managing Directors / Geschäftsführung:
Professor Dr. Paolo Giubellino, Dr. Katharina Stummeyer, Jörg Blaurock
Chairman of the Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:
Ministerialdirigent Dr. Volkmar Dietz

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org




--

Thomas Roth
Department: Informationstechnologie
Location: SB3 2.291


GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de

Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528
Managing Directors / Geschäftsführung:
Professor Dr. Paolo Giubellino, Dr. Katharina Stummeyer, Jörg Blaurock
Chairman of the Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:
Ministerialdirigent Dr. Volkmar Dietz

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] nodemap exports

2024-06-18 Thread Thomas Roth via lustre-discuss

OK, client 247 is the good one ;-)

It is rather an LNET issue.

All three clients have:

> options lnet networks="o2ib8(ib0),o2ib6(ib0)"

I can mount my nodemapped Lustre, with its  MGS on 10.20.6.63@o2ib8, and 
another Lustre on o2ib6.

However, the log of the MGS at 10.20.6.63@o2ib8 remarks
> MGS: Connection restored to 15da687f-7a11-4913-9f9d-3c764c97a9cb (at 
10.20.3.246@o2ib6)
> MGS: Connection restored to 059da3e0-b277-4767-8678-e83730769fb8 (at 
10.20.3.248@o2ib6)
and then
> MGS: Connection restored to a0ee6b8c-48b9-46e8-ba2c-9448889c77ed (at 
10.20.3.247@o2ib8)

I can see the "alien" nids also in
mgs # ls /proc/fs/lustre/mgs/MGS/exports
... 10.20.3.246@o2ib6  10.20.3.247@o2ib8  10.20.3.248@o2ib6

Question is: Why would an MGS on "o2ib8" accept connections from a client on 
"o2ib6"?

Obviously, this would not happen if the client had only o2ib6. So the MGS is somewhat confused, since 
the LNET connection is actually via o2ib8, but the labels and the nodemapping use o2ib6.

This is boot-resistant, the MGS has these wrong nids stored somewhere - can I erase them and start 
again with correct nids?

Regards
Thomas

On 6/18/24 17:33, Thomas Roth via lustre-discuss wrote:

Hi all,

what is the meaning of the "exports" property/parameter of a nodemap?

I have

mgs ]# lctl nodemap_info newclients
...

There are three clients:

mgs # lctl get_param nodemap.newclients.ranges
nodemap.newclients.ranges=
[
  { id: 13, start_nid: 10.20.3.246@o2ib8, end_nid: 10.20.3.246@o2ib8 },
  { id: 12, start_nid: 10.20.3.247@o2ib8, end_nid: 10.20.3.247@o2ib8 },
  { id: 9, start_nid: 10.20.3.248@o2ib8, end_nid: 10.20.3.248@o2ib8 }

This nodemap has

nodemap.newclients.admin_nodemap=0
nodemap.newclients.trusted_nodemap=0
nodemap.newclients.deny_unknown=1

and

mgs # lctl get_param nodemap.newclients.exports
nodemap.newclients.exports=
[
  { nid: 10.20.3.247@o2ib8, uuid: 5d9964f9-81eb-4ea5-93dc-a145534f9e74 },
]

_This_ client, 10.20.3.247, behaves differently: No access for root (ok!), no 
access for a regular user.

The other two clients, 10.20.3.246/248, show no access for root (ok!), while a regular users sees the 
squashed (uid 99) directories of the top level and his own directories/files with correct uid/gid 
beneath.

And the only difference between these clients seems to be these 'exports' (totally absent from the 
manual, btw).

Regards,
Thomas

--

Thomas Roth
Department: Informationstechnologie
Location: SB3 2.291

GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de

Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528
Managing Directors / Geschäftsführung:
Professor Dr. Paolo Giubellino, Dr. Katharina Stummeyer, Jörg Blaurock
Chairman of the Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:
Ministerialdirigent Dr. Volkmar Dietz

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] nodemap exports

2024-06-18 Thread Thomas Roth via lustre-discuss


Hi all,

what is the meaning of the "exports" property/parameter of a nodemap?

I have

mgs ]# lctl nodemap_info newclients
...

There are three clients:

mgs # lctl get_param nodemap.newclients.ranges
nodemap.newclients.ranges=
[
 { id: 13, start_nid: 10.20.3.246@o2ib8, end_nid: 10.20.3.246@o2ib8 },
 { id: 12, start_nid: 10.20.3.247@o2ib8, end_nid: 10.20.3.247@o2ib8 },
 { id: 9, start_nid: 10.20.3.248@o2ib8, end_nid: 10.20.3.248@o2ib8 }


This nodemap has

nodemap.newclients.admin_nodemap=0
nodemap.newclients.trusted_nodemap=0
nodemap.newclients.deny_unknown=1

and

mgs # lctl get_param nodemap.newclients.exports
nodemap.newclients.exports=
[
 { nid: 10.20.3.247@o2ib8, uuid: 5d9964f9-81eb-4ea5-93dc-a145534f9e74 },
]



_This_ client, 10.20.3.247, behaves differently: No access for root (ok!), no 
access for a regular user.

The other two clients, 10.20.3.246/248, show no access for root (ok!), while a regular users sees the 
squashed (uid 99) directories of the top level and his own directories/files with correct uid/gid beneath.



And the only difference between these clients seems to be these 'exports' (totally absent from the 
manual, btw).



Regards,
Thomas


--

Thomas Roth
Department: Informationstechnologie
Location: SB3 2.291


GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de

Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528
Managing Directors / Geschäftsführung:
Professor Dr. Paolo Giubellino, Dr. Katharina Stummeyer,Jörg Blaurock
Chairman of the Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:
Ministerialdirigent Dr. Volkmar Dietz

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] default Nodemap: ll_close_inode_openhandle errors

2024-04-23 Thread Thomas Roth via lustre-discuss


Hi all,

as described before, I have a test cluster with the nodemap feature activated, and managed to get a bunch of clients to mount (by deactivating 
selinux), while these clients are in the "default" nodemap.

I left the "default"s property `admin=0` and set `trusted=1`, and everything 
seemed to work.

Now I ran a compile+install benchmark on these nodes, which finished 
successfully everywhere and with the expected performance.

However, the client logs all show a large number of the following errors:

> LustreError: 2967:0:(mdc_locks.c:1388:mdc_intent_getattr_async_interpret()) mdstest-MDT-mdc-9a0bc355f000: ldlm_cli_enqueue_fini() failed: 
rc = -13


> LustreError: 53842:0:(file.c:241:ll_close_inode_openhandle()) mdstest-clilmv-9a0bc355f000: inode [0x20429:0x9d8d:0x0] mdc close failed: rc 
= -13



Error code 13 is /* Permission denied */

Therefore I repeated the benchmark on one of the "Admin" nodes - the nodemap with both `admin=1` and `trusted=1` - and this client does not show these 
errors.


More than coincidence?

Given that the benchmark seems to have finished successfully, I would ignore these errors. On the other hand, if inodes cannot be handled - that 
sounds severe?


Regards,
Thomas
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] default Nodemap : clients cannot mount

2024-04-22 Thread Thomas Roth via lustre-discuss

OK - my bad: selinux was on.

It's a bunch of test hosts = sloppy configuration = default selinux settings.

With selinux=disabled, one of these hosts mounts, and if I give it trusted=1, the users are enabled,
root is squashed - all fine.

Cheers
Thomas

On 4/22/24 16:50, Thomas Roth via lustre-discuss wrote:

Hi all,

- Lustre 2.15.4 test system with MDS + 2 OSS + 2 administrative clients.

I activated nodemapping and put all these hosts into an "Admin" nodemap, which has the properties
`admin=1` and `trusted=1` - all works fine.

Now there are a couple of other hosts which should not become administrative clients, but just
standard clients => they should be / remain in the "default" nodemap.

The "default" nodemap has `admin=0` and `trusted=0`, as verified by
`lctl get_param nodemap.default`

- and these hosts cannot mount.
Error message is "l_getidentity: no such user 99"

I verified that these hosts actually are seen as "default" nodes by setting `admin=1` for one of them
- mounts.

Umount, lustre_rmmod, set `admin=0` again - does not mount anymore.

Atm I do not see what I overlooked, but I am certain this has worked before, where "before" would mean
other hardware and perhaps Lustre 2.15.1

*Switching* is still ok:
- Put client X into "Admin" nodemap, mount Lustre, remove client X from "Admin" nodemap, wait, try to
`ls` as root - fails.

Set the property `trusted=1` on the "default" nodemap, wait, try to `ls` as a
user - works.

However, this defeats the purpose of having a usable default...

Regards,
Thomas

Thomas Roth
Department: Informationstechnologie
Location: SB3 2.291

GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de

Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528
Managing Directors / Geschäftsführung:
Professor Dr. Paolo Giubellino, Jörg Blaurock
Chairman of the Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:
Ministerialdirigent Dr. Volkmar Dietz

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] default Nodemap : clients cannot mount