I ended up installing lustre 2.15.1 on the new servers. Now compression works
as expected when PFL is enabled.
What I have tested so far is:
AlmaLinux 9.4 - Lustre 2.16.1 - ZFS 2.1.16 - No compression with PFL
AlmaLinux 8.10 - Lustre 2.15.7 - ZFS 2.2.8 - No compression with PFL
AlmaLinux 8.10 - Lustre 2.15.5 - ZFS 2.1.15 - No compression with PFL
AlmaLinux 8.6 - Lustre 2.15.1 - ZFS 2.1.5 - PFL + compression works as expected
I tried installing 2.15.2, 2.15.3 and 2.15.4 but that failed with ksym errors.
It seems that somewhere between lustre 2.15.1 and 2.15.5 the PFL + compression
breaks.
gr,
Martin Balvers
AlmaLinux 8.6 - Lustre 2.15.1 - ZFS 2.1.5
[root@microbio pfl]# ls -lh *
-rw-r--r-- 1 root root 6.3G Jul 1 13:28 test1.txt
-rw-r--r-- 1 root root 6.3G Jul 23 09:34 test2.txt
-rw-r--r-- 1 root root 6.3G Jul 23 09:35 test3.txt
LUSTRE:SVNAME COMPRESS NAME RATIO REFRATIO USED LUSED
lustre-MDT0000 zstd-fast mdt/mdt 1.51x 1.51x 8.23M 6.80M
lustre-OST0000 zstd lustre/ost 4.60x 4.60x 1.37G 6.29G
lustre-OST0001 zstd lustre/ost 4.60x 4.60x 1.37G 6.29G
lustre-OST0002 zstd lustre/ost 4.60x 4.60x 1.37G 6.29G
[root@microbio pfl]# lfs getstripe test1.txt
test1.txt
lcm_layout_gen: 40
lcm_mirror_count: 1
lcm_entry_count: 5
lcme_id: 1
lcme_mirror_id: 0
lcme_flags: init
lcme_extent.e_start: 0
lcme_extent.e_end: 1048576
lmm_stripe_count: 0
lmm_stripe_size: 1048576
lmm_pattern: mdt
lmm_layout_gen: 0
lmm_stripe_offset: 0
lcme_id: 2
lcme_mirror_id: 0
lcme_flags: init
lcme_extent.e_start: 1048576
lcme_extent.e_end: 134217728
lmm_stripe_count: 1
lmm_stripe_size: 4194304
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 0
lmm_objects:
- 0: { l_ost_idx: 0, l_fid: [0x100000000:0x6e:0x0] }
lcme_id: 3
lcme_mirror_id: 0
lcme_flags: init
lcme_extent.e_start: 134217728
lcme_extent.e_end: 2147483648
lmm_stripe_count: 2
lmm_stripe_size: 4194304
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 2
lmm_objects:
- 0: { l_ost_idx: 2, l_fid: [0x100020000:0x6e:0x0] }
- 1: { l_ost_idx: 1, l_fid: [0x100010000:0x6f:0x0] }
lcme_id: 5
lcme_mirror_id: 0
lcme_flags: init
lcme_extent.e_start: 2147483648
lcme_extent.e_end: 6979321856
lmm_stripe_count: 3
lmm_stripe_size: 4194304
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 2
lmm_objects:
- 0: { l_ost_idx: 2, l_fid: [0x100020000:0x6f:0x0] }
- 1: { l_ost_idx: 1, l_fid: [0x100010000:0x70:0x0] }
- 2: { l_ost_idx: 0, l_fid: [0x100000000:0x6f:0x0] }
lcme_id: 6
lcme_mirror_id: 0
lcme_flags: extension
lcme_extent.e_start: 6979321856
lcme_extent.e_end: EOF
lmm_stripe_count: 0
lmm_extension_size: 268435456
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: -1
________________________________
From: lustre-discuss <[email protected]> on behalf of
BALVERS Martin via lustre-discuss <[email protected]>
Sent: Tuesday, July 29, 2025 09:19
To: Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]
<[email protected]>; lustre-discuss <[email protected]>
Subject: Re: [lustre-discuss] [EXTERNAL] [BULK] No ZFS compression on OST's
when using PFL
WARNING - EXTERNAL SENDER - BE CYBERSAFE
Hi,
My initial copy from old lustre to new lustre was with rsync, using this syntax:
SOURCE="/lustre/workgroups" DEST="${SOURCE#/lustre/}" && find "$SOURCE"
-maxdepth 1 -mindepth 1 -type d | xargs -t -n1 -P6 -I% rsync -aPA
--delete-during % "/mnt/lustre_new/$DEST"
I do indeed need to copy the linux ACL's
But when debugging the issue, I used files without any extended attributes,
from other filesystems than lustre.
Just enabling DoM with "lfs setstripe -E 1M -L mdt -E -1 -c 1
/mnt/lustre_new/dom" will disable compression on the OST's. (dom_stripesize is
set to 1048576)
I'm thinking about trying 2.15.1 on the new servers...
Regards,
Martin Balvers
________________________________
From: Vicker, Darby J. (JSC-EG111)[Jacobs Technology, Inc.]
<[email protected]>
Sent: Monday, July 28, 2025 16:57
To: BALVERS Martin <[email protected]>; lustre-discuss
<[email protected]>
Subject: Re: [EXTERNAL] [BULK] [lustre-discuss] No ZFS compression on OST's
when using PFL
WARNING - EXTERNAL SENDER - BE CYBERSAFE
I’m not sure if this is related to your problem but you need to be careful when
copying data between two lustre filesystems. If you use something that copies
the extended attributes (e.g. rsync -X), you can copy the lustre xattrs, which
is probably not what you want. This will override the PFL you have on the new
lustre FS. You can either not copy the extended attributes (which isn’t a
great option if you are using ACL’s or other xattrs) or use a lustre-aware tool
like mpifileutils.
From: lustre-discuss <[email protected]> on behalf of
BALVERS Martin via lustre-discuss <[email protected]>
Date: Friday, July 25, 2025 at 7:07 AM
To: lustre-discuss <[email protected]>
Subject: [EXTERNAL] [BULK] [lustre-discuss] No ZFS compression on OST's when
using PFL
CAUTION: This email originated from outside of NASA. Please take care when
clicking links or opening attachments. Use the "Report Message" button to
report suspicious messages to the NASA SOC.
Hi,
I am using a lustre 2.15.1 cluster with PFL. I'm using the following layout:
lfs setstripe -E 1M -L mdt -E 128M -c 1 -S 4M -E 2G -c 2 -z 64M -E -1 -c -1 -z
256M /lustre
OST's have zpools with zstd compression enabled.
This has worked fine for me for many years.
I now have new servers and installed 2.16.1 on them. I used the same PFL
layout, and after some benchmarking I proceeded to copy data from the old
lustre to the new.
After a while I noticed that it was filling up faster that expected, and say
that the compression ratio on the zpools was 1.
Compression on the zpools is enabled, it should have worked as far as I can
tell.
When I disable PFL, compression works as expected.
I have attached the some results in a txt file.
I have also tried 2.15.7 with the same results. Enabling PFL, or just DoM
disables compression on the OST's.
Am I missing something obvious here?
Regards,
Martin Balvers
Ce message électronique et tous les fichiers attachés qu'il contient sont
confidentiels et destinés exclusivement à l'usage de la personne à laquelle ils
sont adressés. Si vous avez reçu ce message par erreur, merci de le retourner à
son émetteur. Les idées et opinions présentées dans ce message sont celles de
son auteur, et ne représentent pas nécessairement celles de DANONE ou d'une
quelconque de ses filiales. La publication, l'usage, la distribution,
l'impression ou la copie non autorisée de ce message et des attachements qu'il
contient sont strictement interdits.
This e-mail and any files transmitted with it are confidential and intended
solely for the use of the individual to whom it is addressed. If you have
received this email in error please send it back to the person that sent it to
you. Any views or opinions presented are solely those of its author and do not
necessarily represent those of DANONE or any of its subsidiary companies.
Unauthorized publication, use, dissemination, forwarding, printing or copying
of this email and its associated attachments is strictly prohibited.
_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
