hi,
i guess that's kernel version compatible. do you try with kernel
6.8.0-38 ?
夏天
[email protected]
Original
From:lustre-discuss-request <[email protected]>
Sent Time:2025-02-8- 02:27
To:lustre-discuss <[email protected]>
Subject:lustre-discuss Digest, Vol 227, Issue 4
Send lustre-discuss mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of lustre-discuss digest..."
Today's Topics:
1. Problem with 2.16.1 server on Ubuntu Noble 24.04 (?ke Sandgren)
2. Re: Lnet not going up with InfiniHost III Lx HCA card
(Jesse Stroik)
3. Re: Lustre and ZFS draid (Cameron Harr)
----------------------------------------------------------------------
Message: 1
Date: Fri, 7 Feb 2025 15:43:02 +0000
From: ?ke Sandgren
To: "[email protected]"
Subject: [lustre-discuss] Problem with 2.16.1 server on Ubuntu Noble
24.04
Message-ID:
<gvzp280mb0652be277ccafed07f805377e8...@gvzp280mb0652.swep280.prod.outlook.com>
Content-Type: text/plain; charset="iso-8859-1"
Hi!
I'm trying to get 2.16.1 working on Ubuntu 24.04 (kernel 6.8.0-52-generic
Ubuntu)
I have everything built according to the procedure I used when doing this with
2.14 on Ubuntu 20.04 and 22.04.
I.e. after a bit of updating ldiskfs patches I have everything built cleanly.
However, when I try to load lnet it fails and I get this back:
===
[Fri Feb 7 16:10:28 2025] BPF: type_id=13 bits_offset=256
[Fri Feb 7 16:10:28 2025] BPF:
[Fri Feb 7 16:10:28 2025] BPF: Invalid name
[Fri Feb 7 16:10:28 2025] BPF:
[Fri Feb 7 16:10:28 2025] failed to validate module [libcfs] BTF: -22
===
Any ideas for what I managed to mess up?
I can't find anything relevant in master branch that might fix this, so did I
mess up my kernel build or is this a new problem?
---
Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden
Internet: [email protected] ?Mobile: +46 70 7716134 ?Fax: +46 90-580 14
WWW: http://www.hpc2n.umu.se
------------------------------
Message: 2
Date: Fri, 7 Feb 2025 18:24:16 +0000
From: Jesse Stroik
To: Ramiro Alba Queipo ,
"[email protected]"
Subject: Re: [lustre-discuss] Lnet not going up with InfiniHost III Lx
HCA card
Message-ID:
<ph0pr06mb77218abe432182a84cb31cf2da...@ph0pr06mb7721.namprd06.prod.outlook.com>
Content-Type: text/plain; charset="iso-8859-1"
Hi Ramiro,
The invalid MR size looks like you're running into a limit with your cards
setting up the RDMA (o2ib) LND when bringing up the network. There may be
adjustments or workarounds for it possibly including setting map_on_demand=0 as
an argument to the lnet module there.
And since you are using older IB hardware on a newer OS, just a heads up: we
recently ran into an issue with connectx-3 IB cards after upgrading our
operating systems where we found RMDA communication to be unreliable possibly
because they often would exceed the amount of connection queue pairs they could
create. For us, the workaround was to use the ksocklnd instead of o2iblnd. If
you have trouble getting the o2ib lustre network driver to work with this older
hardware due to RDMA problems, that could be a workaround although it may not
be feasible to implement depending on your networking setup.
Best,
Jesse
________________________________________
From: lustre-discuss <[email protected]> on behalf of
Ramiro Alba Queipo
Sent: Thursday, February 6, 2025 3:34 AM
To: [email protected]
Subject: [lustre-discuss] Lnet not going up with InfiniHost III Lx HCA card
Hi all,
I am testing Ubuntu 24.04 (6.8.0-52-generic) client with Lustre 2.16.1 over
Infiniband and using an old Mellanox DDR card (InfiniHost III Lx HCA).
- # ip -br a
options lnet networks=o2ib0(ib0)
- # modprobe lnet
- # lctl network up
LNET configure error 100: Network is down
- # tail -10 /var/log/kernel.log
LNetError: 5071:0:(o2iblnd.c:2866:kiblnd_hdev_get_attr()) Invalid mr size:
0xffffffffffffffff
LNetError: 5071:0:(o2iblnd.c:3103:kiblnd_dev_failover()) Can't get device
attributes: -22
LNetError: 5071:0:(o2iblnd.c:3831:kiblnd_startup()) ko2iblnd: Can't initialize
device: rc = -22
LNetError: Error -100 starting up LNI o2ib
Lustre 2.15.0 and Ubuntu 20.04 (kernel 5.4.0-198-generic) is working fine with
the same hardware
Can anyone give me some advice or idea to make it work?
Thans in advance
Best regards
--
Ramiro Alba
Centre Tecnol?gic de Tranfer?ncia de Calor
http://www.cttc.upc.edu
Escola T?cnica Superior d'Enginyeries
Industrial i Aeron?utica de Terrassa
Colom 11, E-08222, Terrassa, Barcelona, Spain
Tel: (+34) 93 739 8928
------------------------------
Message: 3
Date: Fri, 7 Feb 2025 10:26:49 -0800
From: Cameron Harr
To: [email protected]
Subject: Re: [lustre-discuss] Lustre and ZFS draid
Message-ID: <[email protected]>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
We've been using draid in production since 2020 and I think were
generally happy with it. We have quite a few Lustre clusters and on the
majority of them, we run 90-drive JBODs with 1 OST/OSS node, 1 OST/pool
and 1 pool/JBOD. We use a draid2:8d:90c:2s config and let the
distributed spares rebuild (~2-4 hours) before replacing the 16TB
physical disk, which then rebuilds within a day or so.
An important note with this configuration is that we also include NVMe
in the pool as special allocation devices configured to store small
blocks up to 16K. We probably have much more NVMe space than we need due
to large NVMe drives (zpool list -v shows each mirror capacity is still
low), but we're happy with performance.
NAME STATE READ WRITE CKSUM
asp8 ONLINE 0 0 0
draid2:8d:90c:2s-0 ONLINE 0 0 0
L0 ONLINE 0 0 0
L1 ONLINE 0 0 0
...
L88 ONLINE 0 0 0
L89 ONLINE 0 0 0
special
mirror-1 ONLINE 0 0 0
N6 ONLINE 0 0 0
N7 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
N8 ONLINE 0 0 0
N9 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
N10 ONLINE 0 0 0
N11 ONLINE 0 0 0
spares
draid2-0-0 AVAIL
draid2-0-1 AVAIL
On our newest systems, we have some 106-drive JBODs with 20TB drives and
in order to reduce the chance of multiple disk failures in a single
draid device, we reconfigured the pools to have 2 draid devices per
pool, though still one OST per pool and one OST per OSS. In this config
we only have one distributed spare per draid. Due to significant write
performance reasons we also (reluctantly) started spanning pools across
2 JBODs. An additional difference is we had much less NVMe capacity on
these systems with just one small pair of NVMe drives per enclosure, so
we configure them as special devices for pool metadata rather than for
small block storage. The config for one of those pools looks like the
following:
NAME STATE READ WRITE CKSUM
merced239 ONLINE 0 0 0
draid2:11d:53c:1s-0 ONLINE 0 0 0
L0 ONLINE 0 0 0
L2 ONLINE 0 0 0
L4 ONLINE 0 0 0
...
L100 ONLINE 0 0 0
L102 ONLINE 0 0 0
L104 ONLINE 0 0 0
draid2:11d:53c:1s-1 ONLINE 0 0 0
U1 ONLINE 0 0 0
U3 ONLINE 0 0 0
U5 ONLINE 0 0 0
...
U101 ONLINE 0 0 0
U103 ONLINE 0 0 0
U105 ONLINE 0 0 0
special
mirror-2 ONLINE 0 0 0
N2 ONLINE 0 0 0
N3 ONLINE 0 0 0
spares
draid2-0-0 AVAIL
draid2-1-0 AVAIL
Hope this helps,
Cameron
On 2/6/25 11:29 AM, Nehring, Shane R [ITS] wrote:
> Hello All,
>
> I didn't want to hijack the other thread today about draid, but I have been
> meaning to ask questions about it and folks' experience with it in the
context
> of Lustre. Most of my questions come from not having a chance to really
play
> around with draid much.
>
> Have you been generally satisfied with performance of a single draid vdev
vs
> either multiple pools/osts per node or single osts on a pool spanning
multiple
> raidz(2) vdev members? Is random io comparable to a span of raidz2 vdevs?
I know
> one of the pain points (more from a space usage perspective as I
understand it)
> is the fixed stripe width and how that impacts small files, but does small
file
> io perform particularly badly on draid vs a span raidz2?
>
> I've got hardware on order (a couple 60 bay jbods and heads) that's going
to
> replace some of the older OSTs in our current volume and I'm leaning
toward a
> single draid pool OST per OSS. I plan to do some benchmarking of the pools
in
> various configurations, but it's hard to generate a benchmark that's
actually
> representative of real world usage.
>
> If you've got any insights or anecdotes regarding your experience with
draid and
> Lustre I'd love to hear them!
>
> Thanks,
> Shane
>
> _______________________________________________
> lustre-discuss mailing list
> [email protected]
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
------------------------------
Subject: Digest Footer
_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
------------------------------
End of lustre-discuss Digest, Vol 227, Issue 4
**********************************************_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
